{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DQN (Deep Q-Network)\n",
    "### Function approximation Q-learning\n",
    "This tutorial walks through the implementation of deep Q networks (DQNs), \n",
    "an RL method which applies the function approximation capabilities of deep neural networks\n",
    "to problems in reinforcement learning.\n",
    "The model in this tutorial closely follows the work described in the paper \n",
    "[Human-level control through deep reinforcement learning](http://www.nature.com/nature/journal/v518/n7540/full/nature14236.html?foxtrotcallback=true), written by Volodomyr Mnih. \n",
    "\n",
    "To keep these chapters runnable \n",
    "by as many people as possible, \n",
    "on as many machines as possible,\n",
    "and with as few headaches as possible, \n",
    "we have so far avoided dependencies on external libraries \n",
    "(besides mxnet, numpy and matplotlib). \n",
    "However, in this case, we'll need to import the [OpenAI Gym](https://gym.openai.com/docs).\n",
    "That's because in reinforcement learning, \n",
    "instead of drawing examples from a data structure, \n",
    "our data comes from interactions with an environment. \n",
    "In this chapter, our environemnts will be classic Atari video games.\n",
    "\n",
    "## Preliminaries\n",
    "The following code clones and installs the OpenAI gym.\n",
    "`git clone https://github.com/openai/gym ; cd gym ; pip install -e .[all]` \n",
    "Full documentation for the gym can be found on [at this website](https://gym.openai.com/).\n",
    "If you want to see reasonable results before the sun sets on your AI career,\n",
    "we suggest running these experiments on a server equipped with GPUs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/matplotlib/__init__.py:1067: UserWarning: Duplicate key in file \"/home/ubuntu/.config/matplotlib/matplotlibrc\", line #2\n",
      "  (fname, cnt))\n",
      "/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/matplotlib/__init__.py:1067: UserWarning: Duplicate key in file \"/home/ubuntu/.config/matplotlib/matplotlibrc\", line #3\n",
      "  (fname, cnt))\n"
     ]
    }
   ],
   "source": [
    "import mxnet as mx\n",
    "from mxnet import nd, autograd\n",
    "from mxnet import gluon\n",
    "from __future__ import print_function\n",
    "import os\n",
    "import random\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from IPython import display\n",
    "import gym\n",
    "import math\n",
    "from collections import namedtuple\n",
    "import time\n",
    "f = open('results.txt','w')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary of the algorithm\n",
    "#### Collect samples\n",
    "At the beginning of each episode (one round of the game), \n",
    "reset the environment to its initial state using `env.reset()`. \n",
    "At each time step ``t``, the environment is at `current_state`.\n",
    "With probability $\\epsilon$, apply a random action.\n",
    "Otherwise, apply $argmax_a~ Q(\\phi($ `current_state` $),a,\\theta)$,\n",
    "where $Q$ is parameterized by paramters $\\theta$ and $\\phi(\\cdot)$ is preprocessor.\n",
    "Pass the action through `env.step(action)` to receive next frame, reward and whether the game terminates.\n",
    "Append this frame to the end of the `current_state` and construct `next_state` while removeing $frame(t-12)$.\n",
    "Store the tuple $(\\phi($ `current_state` $), action, reward, \\phi($ `next_ state` $))$ in the replay buffer.\n",
    "\n",
    "#### Update Network\n",
    "* Draw batches of tuples from the replay buffer: $(\\phi,r,a,\\phi')$.\n",
    "* Define the following loss\n",
    "$$\\Large(\\small Q(\\phi,a,\\theta)-r-argmax_{a'}Q(\\phi',a',\\theta^-)\\Large)^2$$\n",
    "* Where $\\theta^-$ is the parameter of the target network.( Set $Q(\\phi',a',\\theta^-)$ to zero if $\\phi$ is the preprocessed termination state). \n",
    "* Update the $\\theta$\n",
    "* Update the $\\theta^-$ once in a while\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Set the hyper-parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ubuntu/gym/gym/envs/registration.py:14: DeprecationWarning: Parameters to load are deprecated.  Call .resolve and .require separately.\n",
      "  result = entry_point.load(False)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "batch_size: 32, image_size: 84, replay_buffer_size: 1000000, learning_frequency: 4, skip_frame: 4, internal_skip_frame: 4, frame_len: 4, Target_update: 10000, epsilon_min: 0.1, annealing_end: 1000000.0, gamma: 0.99, replay_start_size: 50000, no_op_max: 7.5, num_episode: 150, lr: 0.00025, gamma1: 0.95, gamma2: 0.95, rms_eps: 0.01, ctx: gpu(0)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ubuntu/gym/gym/__init__.py:22: UserWarning: DEPRECATION WARNING: to improve load times, gym no longer automatically loads gym.spaces. Please run \"import gym.spaces\" to load gym.spaces on your own. This warning will turn into an error in a future version of gym.\n",
      "  warnings.warn('DEPRECATION WARNING: to improve load times, gym no longer automatically loads gym.spaces. Please run \"import gym.spaces\" to load gym.spaces on your own. This warning will turn into an error in a future version of gym.')\n"
     ]
    }
   ],
   "source": [
    "class Options:\n",
    "    def __init__(self):\n",
    "        #Architecture\n",
    "        self.batch_size = 32 # The size of the batch to learn the Q-function\n",
    "        self.image_size = 84 # Resize the raw input frame to square frame of size 80 by 80 \n",
    "        #Trickes\n",
    "        self.replay_buffer_size = 1000000 # The size of replay buffer; set it to size of your memory (.5M for 50G available memory)\n",
    "        self.learning_frequency = 4 # With Freq of 1/4 step update the Q-network\n",
    "        self.skip_frame = 4 # Skip 4-1 raw frames between steps\n",
    "        self.internal_skip_frame = 4 # Skip 4-1 raw frames between skipped frames\n",
    "        self.frame_len = 4 # Each state is formed as a concatination 4 step frames [f(t-12),f(t-8),f(t-4),f(t)]\n",
    "        self.Target_update = 10000 # Update the target network each 10000 steps\n",
    "        self.epsilon_min = 0.1 # Minimum level of stochasticity of policy (epsilon)-greedy\n",
    "        self.annealing_end = 1000000. # The number of step it take to linearly anneal the epsilon to it min value\n",
    "        self.gamma = 0.99 # The discount factor\n",
    "        self.replay_start_size = 50000 # Start to backpropagated through the network, learning starts\n",
    "        self.no_op_max = 30 / self.skip_frame # Run uniform policy for first 30 times step of the beginning of the game\n",
    "        \n",
    "        #otimization\n",
    "        self.num_episode = 150 # Number episode to run the algorithm\n",
    "        self.lr = 0.00025 # RMSprop learning rate\n",
    "        self.gamma1 = 0.95 # RMSprop gamma1\n",
    "        self.gamma2 = 0.95 # RMSprop gamma2\n",
    "        self.rms_eps = 0.01 # RMSprop epsilon bias\n",
    "        \n",
    "        # Enables gpu if available, if not, set it to mx.cpu()\n",
    "        gpus = mx.test_utils.list_gpus()\n",
    "        context = mx.gpu() if len(gpus) > 0 else mx.cpu()\n",
    "        self.ctx = context\n",
    "opt = Options()\n",
    "\n",
    "env_name = 'AssaultNoFrameskip-v4' # Set the desired environment\n",
    "env = gym.make(env_name)\n",
    "num_action = env.action_space.n # Extract the number of available action from the environment setting\n",
    "\n",
    "manualSeed = 1 # random.randint(1, 10000) # Set the desired seed to reproduce the results\n",
    "mx.random.seed(manualSeed)\n",
    "attrs = vars(opt)\n",
    "print (', '.join(\"%s: %s\" % item for item in attrs.items()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define the DQN model\n",
    "The network is constructed as three CNN layers and a fully connected added on the top. Furthermore, the optimizer is assigned to the parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "DQN = gluon.nn.Sequential()\n",
    "with DQN.name_scope():\n",
    "    #first layer\n",
    "    DQN.add(gluon.nn.Conv2D(channels=32, kernel_size=8,strides = 4,padding = 0))\n",
    "    DQN.add(gluon.nn.BatchNorm(axis = 1, momentum = 0.1,center=True))\n",
    "    DQN.add(gluon.nn.Activation('relu'))\n",
    "    #second layer\n",
    "    DQN.add(gluon.nn.Conv2D(channels=64, kernel_size=4,strides = 2))\n",
    "    DQN.add(gluon.nn.BatchNorm(axis = 1, momentum = 0.1,center=True))\n",
    "    DQN.add(gluon.nn.Activation('relu'))\n",
    "    #tird layer\n",
    "    DQN.add(gluon.nn.Conv2D(channels=64, kernel_size=3,strides = 1))\n",
    "    DQN.add(gluon.nn.BatchNorm(axis = 1, momentum = 0.1,center=True))\n",
    "    DQN.add(gluon.nn.Activation('relu'))\n",
    "    DQN.add(gluon.nn.Flatten())\n",
    "    #fourth layer\n",
    "    DQN.add(gluon.nn.Dense(512,activation ='relu'))\n",
    "    #fifth layer\n",
    "    DQN.add(gluon.nn.Dense(num_action,activation ='relu'))\n",
    "\n",
    "dqn = DQN\n",
    "dqn.collect_params().initialize(mx.init.Normal(0.02), ctx=opt.ctx)\n",
    "DQN_trainer = gluon.Trainer(dqn.collect_params(),'RMSProp', \\\n",
    "                          {'learning_rate': opt.lr ,'gamma1':opt.gamma1,'gamma2': opt.gamma2,'epsilon': opt.rms_eps,'centered' : True})\n",
    "dqn.collect_params().zero_grad()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "Target_DQN = gluon.nn.Sequential()\n",
    "with Target_DQN.name_scope():\n",
    "    #first layer\n",
    "    Target_DQN.add(gluon.nn.Conv2D(channels=32, kernel_size=8,strides = 4,padding = 0))\n",
    "    Target_DQN.add(gluon.nn.BatchNorm(axis = 1, momentum = 0.1,center=True))\n",
    "    Target_DQN.add(gluon.nn.Activation('relu'))\n",
    "    #second layer\n",
    "    Target_DQN.add(gluon.nn.Conv2D(channels=64, kernel_size=4,strides = 2))\n",
    "    Target_DQN.add(gluon.nn.BatchNorm(axis = 1, momentum = 0.1,center=True))\n",
    "    Target_DQN.add(gluon.nn.Activation('relu'))\n",
    "    #tird layer\n",
    "    Target_DQN.add(gluon.nn.Conv2D(channels=64, kernel_size=3,strides = 1))\n",
    "    Target_DQN.add(gluon.nn.BatchNorm(axis = 1, momentum = 0.1,center=True))\n",
    "    Target_DQN.add(gluon.nn.Activation('relu'))\n",
    "    Target_DQN.add(gluon.nn.Flatten())\n",
    "    #fourth layer\n",
    "    Target_DQN.add(gluon.nn.Dense(512,activation ='relu'))\n",
    "    #fifth layer\n",
    "    Target_DQN.add(gluon.nn.Dense(num_action,activation ='relu'))\n",
    "target_dqn = Target_DQN\n",
    "target_dqn.collect_params().initialize(mx.init.Normal(0.02), ctx=opt.ctx)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Replay buffer\n",
    "Replay buffer store the tuple of : `state`, action , `next_state`, reward , done."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "Transition = namedtuple('Transition',('state', 'action', 'next_state', 'reward','done'))\n",
    "class Replay_Buffer():\n",
    "    def __init__(self, replay_buffer_size):\n",
    "        self.replay_buffer_size = replay_buffer_size\n",
    "        self.memory = []\n",
    "        self.position = 0\n",
    "    def push(self, *args):\n",
    "        if len(self.memory) < self.replay_buffer_size:\n",
    "            self.memory.append(None)\n",
    "        self.memory[self.position] = Transition(*args)\n",
    "        self.position = (self.position + 1) % self.replay_buffer_size\n",
    "    def sample(self, batch_size):\n",
    "        return random.sample(self.memory, batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocess frames\n",
    "* Take a frame, average over the `RGB` filter and append it to the `state` to construct `next_state`\n",
    "* Clip the reward\n",
    "* Render the frames"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess(raw_frame, currentState = None, initial_state = False):\n",
    "    raw_frame = nd.array(raw_frame,mx.cpu())\n",
    "    raw_frame = nd.reshape(nd.mean(raw_frame, axis = 2),shape = (raw_frame.shape[0],raw_frame.shape[1],1))\n",
    "    raw_frame = mx.image.imresize(raw_frame,  opt.image_size, opt.image_size)\n",
    "    raw_frame = nd.transpose(raw_frame, (2,0,1))\n",
    "    raw_frame = raw_frame.astype(np.float32)/255.\n",
    "    if initial_state == True:\n",
    "        state = raw_frame\n",
    "        for _ in range(opt.frame_len-1):\n",
    "            state = nd.concat(state , raw_frame, dim = 0)\n",
    "    else:\n",
    "        state = mx.nd.concat(currentState[1:,:,:], raw_frame, dim = 0)\n",
    "    return state\n",
    "\n",
    "def rew_clipper(rew):\n",
    "    if rew>0.:\n",
    "        return 1.\n",
    "    elif rew<0.:\n",
    "        return -1.\n",
    "    else:\n",
    "        return 0\n",
    "\n",
    "def renderimage(next_frame):\n",
    "    if render_image:\n",
    "        plt.imshow(next_frame);\n",
    "        plt.show()\n",
    "        display.clear_output(wait=True)\n",
    "        time.sleep(.1)\n",
    "        \n",
    "l2loss = gluon.loss.L2Loss(batch_axis=0)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initialize arrays"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "frame_counter = 0. # Counts the number of steps so far\n",
    "annealing_count = 0. # Counts the number of annealing steps\n",
    "epis_count = 0. # Counts the number episodes so far\n",
    "replay_memory = Replay_Buffer(opt.replay_buffer_size) # Initialize the replay buffer\n",
    "tot_clipped_reward = np.zeros(opt.num_episode) \n",
    "tot_reward = np.zeros(opt.num_episode)\n",
    "moving_average_clipped = 0.\n",
    "moving_average = 0."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epis[0],eps[1.000000],durat[288],fnum=287, cum_cl_rew = 8, cum_rew = 168,tot_cl = 0 , tot = 0\n",
      "epis[10],eps[1.000000],durat[230],fnum=2757, cum_cl_rew = 6, cum_rew = 126,tot_cl = 0 , tot = 0\n",
      "epis[20],eps[1.000000],durat[208],fnum=5447, cum_cl_rew = 10, cum_rew = 210,tot_cl = 0 , tot = 0\n",
      "epis[30],eps[1.000000],durat[262],fnum=8024, cum_cl_rew = 17, cum_rew = 357,tot_cl = 0 , tot = 0\n",
      "epis[40],eps[1.000000],durat[321],fnum=10961, cum_cl_rew = 9, cum_rew = 189,tot_cl = 0 , tot = 0\n",
      "epis[50],eps[1.000000],durat[164],fnum=13182, cum_cl_rew = 9, cum_rew = 189,tot_cl = 0 , tot = 0\n",
      "epis[60],eps[1.000000],durat[380],fnum=16195, cum_cl_rew = 17, cum_rew = 357,tot_cl = 9 , tot = 199\n",
      "epis[70],eps[1.000000],durat[252],fnum=18370, cum_cl_rew = 9, cum_rew = 189,tot_cl = 9 , tot = 198\n",
      "epis[80],eps[1.000000],durat[225],fnum=21282, cum_cl_rew = 4, cum_rew = 84,tot_cl = 9 , tot = 195\n",
      "epis[90],eps[1.000000],durat[188],fnum=23805, cum_cl_rew = 7, cum_rew = 147,tot_cl = 9 , tot = 191\n",
      "epis[100],eps[1.000000],durat[225],fnum=26237, cum_cl_rew = 8, cum_rew = 168,tot_cl = 8 , tot = 186\n",
      "epis[110],eps[1.000000],durat[205],fnum=28547, cum_cl_rew = 7, cum_rew = 147,tot_cl = 8 , tot = 178\n",
      "epis[120],eps[1.000000],durat[326],fnum=31374, cum_cl_rew = 16, cum_rew = 336,tot_cl = 8 , tot = 176\n",
      "epis[130],eps[1.000000],durat[245],fnum=33852, cum_cl_rew = 6, cum_rew = 126,tot_cl = 8 , tot = 173\n",
      "epis[140],eps[1.000000],durat[307],fnum=36422, cum_cl_rew = 13, cum_rew = 273,tot_cl = 8 , tot = 172\n"
     ]
    }
   ],
   "source": [
    "render_image = False # Whether to render Frames and show the game\n",
    "batch_state = nd.empty((opt.batch_size,opt.frame_len,opt.image_size,opt.image_size), opt.ctx)\n",
    "batch_state_next = nd.empty((opt.batch_size,opt.frame_len,opt.image_size,opt.image_size), opt.ctx)\n",
    "for i in range(opt.num_episode):\n",
    "    cum_clipped_reward = 0\n",
    "    cum_reward = 0\n",
    "    next_frame = env.reset()\n",
    "    state = preprocess(next_frame, initial_state = True)\n",
    "    t = 0.\n",
    "    done = False\n",
    "    \n",
    "\n",
    "    while not done:\n",
    "        previous_state = state\n",
    "        # show the frame\n",
    "        renderimage(next_frame)\n",
    "        sample = random.random()\n",
    "        if frame_counter > opt.replay_start_size:\n",
    "            annealing_count += 1\n",
    "        if frame_counter == opt.replay_start_size:\n",
    "            print('annealing and laerning are started ')\n",
    "            \n",
    "            \n",
    "        \n",
    "        eps = np.maximum(1.-annealing_count/opt.annealing_end,opt.epsilon_min)\n",
    "        effective_eps = eps\n",
    "        if t < opt.no_op_max:\n",
    "            effective_eps = 1.\n",
    "        \n",
    "        # epsilon greedy policy\n",
    "        if sample < effective_eps:\n",
    "            action = random.randint(0, num_action - 1)\n",
    "        else:\n",
    "            data = nd.array(state.reshape([1,opt.frame_len,opt.image_size,opt.image_size]),opt.ctx)\n",
    "            action = int(nd.argmax(dqn(data),axis=1).as_in_context(mx.cpu()).asscalar())\n",
    "        \n",
    "        # Skip frame\n",
    "        rew = 0\n",
    "        for skip in range(opt.skip_frame-1):\n",
    "            next_frame, reward, done,_ = env.step(action)\n",
    "            renderimage(next_frame)\n",
    "            cum_clipped_reward += rew_clipper(reward)\n",
    "            rew += reward\n",
    "            for internal_skip in range(opt.internal_skip_frame-1):\n",
    "                _ , reward, done,_ = env.step(action)\n",
    "                cum_clipped_reward += rew_clipper(reward)\n",
    "                rew += reward\n",
    "                \n",
    "        next_frame_new, reward, done, _ = env.step(action)\n",
    "        renderimage(next_frame)\n",
    "        cum_clipped_reward += rew_clipper(reward)\n",
    "        rew += reward\n",
    "        cum_reward += rew\n",
    "        \n",
    "        # Reward clipping\n",
    "        reward = rew_clipper(rew)\n",
    "        next_frame = np.maximum(next_frame_new,next_frame)\n",
    "        state = preprocess(next_frame, state)\n",
    "        replay_memory.push((previous_state*255.).astype('uint8'),action,(state*255.).astype('uint8'),reward,done)\n",
    "        # Train\n",
    "        if frame_counter > opt.replay_start_size:        \n",
    "            if frame_counter % opt.learning_frequency == 0:\n",
    "                transitions = replay_memory.sample(opt.batch_size)\n",
    "                batch = Transition(*zip(*transitions))\n",
    "                for j in range(opt.batch_size):\n",
    "                    batch_state[j] = nd.array(batch.state[j],opt.ctx).astype('float32')/255.\n",
    "                    batch_state_next[j] = nd.array(batch.next_state[j],opt.ctx).astype('float32')/255.\n",
    "                batch_reward = nd.array(batch.reward,opt.ctx)\n",
    "                batch_action = nd.array(batch.action,opt.ctx).astype('uint8')\n",
    "                batch_done = nd.array(batch.done,opt.ctx)\n",
    "                with autograd.record():\n",
    "                    Q_sp = nd.max(target_dqn(batch_state_next),axis = 1)\n",
    "                    Q_sp = Q_sp*(nd.ones(opt.batch_size,ctx = opt.ctx)-batch_done)\n",
    "                    Q_s_array = dqn(batch_state)\n",
    "                    Q_s = nd.pick(Q_s_array,batch_action,1)\n",
    "                    loss = nd.mean(l2loss(Q_s ,  (batch_reward + opt.gamma *Q_sp)))\n",
    "                loss.backward()\n",
    "                DQN_trainer.step(opt.batch_size)\n",
    "                \n",
    "        \n",
    "\n",
    "        \n",
    "        t += 1\n",
    "        frame_counter += 1\n",
    "        \n",
    "        # Save the model and update Target model\n",
    "        if frame_counter > opt.replay_start_size:\n",
    "            if frame_counter % opt.Target_update == 0 :\n",
    "                check_point = frame_counter / (opt.Target_update *100)\n",
    "                fdqn = './target_%s_%d' % (env_name,int(check_point))\n",
    "                dqn.save_parameters(fdqn)\n",
    "                target_dqn.load_parameters(fdqn, opt.ctx)\n",
    "        if done:\n",
    "            if epis_count % 10. == 0. :\n",
    "                results = 'epis[%d],eps[%f],durat[%d],fnum=%d, cum_cl_rew = %d, cum_rew = %d,tot_cl = %d , tot = %d'\\\n",
    "                  %(epis_count,eps,t+1,frame_counter,cum_clipped_reward,cum_reward,moving_average_clipped,moving_average)\n",
    "                print(results)\n",
    "                f.write('\\n' + results)\n",
    "    epis_count += 1\n",
    "    tot_clipped_reward[int(epis_count)-1] = cum_clipped_reward\n",
    "    tot_reward[int(epis_count)-1] = cum_reward\n",
    "    if epis_count > 50.:\n",
    "        moving_average_clipped = np.mean(tot_clipped_reward[int(epis_count)-1-50:int(epis_count)-1])\n",
    "        moving_average = np.mean(tot_reward[int(epis_count)-1-50:int(epis_count)-1])\n",
    "f.close()\n",
    "from tempfile import TemporaryFile\n",
    "outfile = TemporaryFile()\n",
    "outfile_clip = TemporaryFile()\n",
    "np.save(outfile, moving_average)\n",
    "np.save(outfile_clip, moving_average_clipped)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot the overall performace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running after 150 number of episodes\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XeYVOX1wPHvoQioKILYqBYWQcrSVGRVRFw0tih2YxdkbVET7P0XjUKMGguggKixREHFqFFQlKYIC9KrnWIAURRRQOD8/jh3wrjM7t7dnT7n8zz77MydW87gOmfuW84rqopzzjlXUrVUB+Cccy49eYJwzjkXkycI55xzMXmCcM45F5MnCOecczF5gnDOOReTJwjnnHMxeYJwzjkXkycI55xzMdVIdQBVsfvuu2vz5s1THYZzzmWU6dOnf6uqDcvbL6MTRPPmzSkuLk51GM45l1FE5Ksw+yWsiUlEhovIKhGZG7WtvYh8JCJzROTfIrJLsP0YEZkebJ8uIj0SFZdzzrlwEtkHMQI4tsS2ocCNqtoWeBXoH2z/Fjgx2H4B8GwC43LOORdCwhKEqk4AviuxOQ+YEDweC/QO9v1EVVcE2+cBdUSkVqJic845V75k90HMA04GXgNOB5rE2Kc3MENVN8Y6gYj0BfoCNG3aNEFhOucyza+//sqyZcvYsGFDqkNJG7Vr16Zx48bUrFmzUscnO0FcDPxDRG4DXgc2Rb8oIgcB9wOFpZ1AVZ8AngDo3LmzL2bhnANg2bJl1K1bl+bNmyMiqQ4n5VSVNWvWsGzZMvbdd99KnSOp8yBUdaGqFqpqJ+AF4LPIayLSGOuXOF9VPyvtHM45F8uGDRto0KCBJ4eAiNCgQYMq3VElNUGIyB7B72rArcDg4Hk94E2sA3tyMmNyzmUPTw6/VdV/j4Q1MYnIC0B3YHcRWQbcAewsIlcEu7wCPBU8vhI4ALhdRG4PthWq6qqEBLd8OQwZUvHjjjsOunaNfzzOOZeGEpYgVPXsUl56OMa+fwH+kqhYtrNiBfylgpdThQcegMmTIT8/MXE55zJa9erVadu2LZs3b2bffffl2WefpV69eqXuv3btWp5//nkuv/zyJEYZXm7WYurSBbZurdjPN99A/fpw8smwKjE3Ns65zFanTh1mzpzJ3LlzqV+/Po899liZ+69du5bHH3+8wtfZsmVLZUOskNxMEJWx117w2muWHE47DTZtKv8Y51zO6tq1K8uXL//f84EDB9KlSxfatWvHHXfcAcCNN97IZ599Rn5+Pv379+eDDz7ghBNO+N8xV155JSNGjACstNANN9xAx44defnll+nevTs33HADBx98MHl5eUycODHu7yGjazElXadOMHw4nHMOXH01DB6c6oicc7Fccw3MnBnfc+bnw0MPhdp1y5YtvPfee1xyySUAjBkzhiVLljB16lRUlZNOOokJEyZw3333MXfuXGYGsX7wwQdlnrdBgwbMmDEDgMGDB7N582amTp3KW2+9xV133cW7775b+fcXgyeIijr7bJg1C+6/H9q3h6KiVEfknEsTv/zyC/n5+SxfvpxWrVpxzDHHAJYgxowZQ4cOHQD46aefWLJkSYUn+5555pm/eX7qqacC0KlTJ7788suqv4ESPEFUxj33wNy5dhfRqhV0757qiJxz0UJ+04+3SB/Ezz//TK9evXjssce4+uqrUVVuuukmLrvsst/sX/JDvUaNGmzduvV/z0vOYdhpp51+87xWLatIVL16dTZv3hzHd2K8D6IyqleH556DAw6w/ogvvkh1RM65NLLjjjvyj3/8gwceeIDNmzfTq1cvhg8fzk8//QTA8uXLWbVqFXXr1mXdunX/O65Zs2bMnz+fjRs3snbtWt57771UvQXAE0Tl7borvP46bNlizU7OORelQ4cOtGvXjhdeeIHCwkLOOeccunbtStu2bTnttNNYt24dDRo0oFu3brRp04b+/fvTpEkTzjjjDNq0acMZZ5zxvyapVBHVzC1n1LlzZ035gkH33AO33gpr11rScM6lxIIFC2jVqlWqw0g7sf5dRGS6qnYu71i/g6iqSIafPTu1cTjnXJx5gqiq9u3ttycI51yW8QRRVfvsYzOsZ81KdSTO5bxMbjJPhKr+e3iCqCoRu4vwOwjnUqp27dqsWbPGk0Qgsh5E7dq1K30OnwcRD+3awZNP2oim6tVTHY1zOalx48YsW7aM1atXpzqUtBFZUa6yPEHEQ/v28PPP8Pnn0KJFqqNxLifVrFmz0iunudi8iSke2rWz394P4ZzLIp4g4uGgg6BaNe+HcM5lFU8Q8VC7NrRs6XcQzrms4gkiXnwkk3Muy3iCiJd27eDLL+GHH1IdiXPOxYUniHiJzKieMye1cTjnXJx4gogXH8nknMsyniDipVEjK7nh/RDOuSzhCSJeROwuwu8gnHNZwhNEPLVvb30QUUsGOudcpvIEEU/t2lnJjc8+S3UkzjlXZZ4g4snXhnDOZRFPEPHUurWV3PB+COdcFkhYghCR4SKySkTmRm1rLyIficgcEfm3iOwS9dpNIvKpiCwSkV6Jiiuh6tSxkht+B+GcywLlJggR2VFEbhORJ4PnLUTkhBDnHgEcW2LbUOBGVW0LvAr0D87ZGjgLOCg45nERycyFFXwkk3MuS4S5g3gK2Ah0DZ4vB/5S3kGqOgH4rsTmPGBC8Hgs0Dt4fDLwoqpuVNUvgE+Bg0PEln7at/eSG865rBAmQeyvqgOAXwFU9WdAKnm9eVgyADgdaBI8bgQsjdpvWbAt80RmVHvJDedchguTIDaJSB1AAURkf+yOojIuBi4XkelAXWBTRU8gIn1FpFhEitNyaUEfyeScyxJhlhy9A3gbaCIizwHdgAsrczFVXQgUAohIHnB88NJytt1NADQOtsU6xxPAEwCdO3dOv9XJGzWC3XbzfgjnXMYrN0Go6lgRmQEcijUt/VFVv63MxURkD1VdJSLVgFuBwcFLrwPPi8jfgX2AFsDUylwj5UR8bQjnXFYoNUGISMcSm74JfjcVkaaqOqOsE4vIC0B3YHcRWYbdiewsIlcEu7yCdYCjqvNE5CVgPrAZuEJVt1T0zaSNdu1g2DAruVHNp5o45zJTWXcQDwS/awOdgVnYHUQ7oJhto5piUtWzS3np4VL2vwe4p6xzZoz27WH9evj8czjggFRH45xzlVLq11tVPUpVj8LuHDqqamdV7QR0oJT+AReIjGTyZibnXAYL0/7RUlX/N2ZTVecCrRIXUhY46CAvueGcy3hhRjHNFpGhwD+D5+cC/tW4LHXqQF6e30E45zJamARxEVAE/DF4PgEYlLCIskX79jA1MwdiOecchGhiUtUNwGPA7cBtwKPBNleWdu3giy8SO6P6ww/hlltA0286iHMu84Up1tcdWAI8CjwOLBaRIxIcV+Y75RSbMNexI9x4o41qiqdff4WLLoJ774UpU+J7buecI1wn9QNAoaoeqapHAL2ABxMbVhZo1QoWLYI//AHuv9/Wihg9On7nHzYMFi+G6tVhkLf4OefiL0yCqKmqiyJPVHUxUDNxIWWRhg3hqadgwgTYZRf4/e/hpJOs2mtV/PQT3HknHH44XHYZvPQSrFkTj4idc+5/wiSIYhEZKiLdg5+h2EQ5F9bhh8OMGTBwIIwbZ4sKNW68/U/TpvBwzHmEv/XAA7ByJQwYAP36wcaNMGJEwt+Gcy63iJbTwSkitYArgIJg00TgcVWtbEXXuOncubMWF2dYrlq6FB58EH78cfvXFi2CSZPg1VftbiOWlSth//3huOPg5ZdtW0GBbV+0yEt7OOfKJSLTVbVzufuVlyBKnLQ+0FhV02KAf0YmiLL88gsceSQsWAAffQRt2my/z+WXw5NPwvz50KKFbXvuOevrGDsWevZMbszOuYwTNkGEGcX0gYjsEiSH6cCTIuKd1IlQp47dPdSta30VJfsVFi2CJ56wfodIcgDo3RsaNPDO6srYutWHCTtXijDtEbuq6o/AqcAzqnoIcHRiw8phjRrBK6/A8uVwxhk2nDXi5pstidx++2+PqV0bLr7YRkkt9zJZFdK7txVUfOutVEfiXNoJkyBqiMjewBnAGwmOxwEceqjdKYwbB3/6k2376CNLHP37wx57bH/MZZfBli02/NWFs24dvPkmLFsGxx8Pp55qfUTOOSBcgrgbeAf4VFWnich+2MQ5l0gXXADXXguPPAJDh1pi2GsvuO662Pvvvz8UFlpi2bw5ubFmqvHj7Q7ttdfgr3+Ft9+2+St/+9tv79ycy1FhSm28rKrtVPXy4Pnnqto78aE5BgywD/2+fWHyZJv7sPPOpe9fVGRNTG++mbQQE2rt2qrPGSnL2LHWZHfUUTbbff58OPpoS8YdO/oMdZfzSh3FJCLXq+oAEXkE2G4nVb060cGVJ+tGMcXy/fdwyCGwww4wcybUKKO+4ubN0Ly5jX56++2khZgw550Hb7wBn35qnfDx1qqV/Xv95z+/3f7663DVVdYE9e23PnTYZZ14jGJaEPwuxkYvlfxxybDbbrauxIcflp0cwF7v0wfeecdWs8tkW7daklu7Fu5JwEKDX38NCxfaHVpJJ50Et95qyfnrr+N/becyRFkryv07+P20qj4NvAq8EvXcJUudOlaqI4xLL7X6TEOGJDamRJs50769N2kCjz5qlXHjaexY+33MMbFfbxWsiTV/fnyv61wGCTMPorOIzMEWCZorIrNEpFPiQ3OV0qgRnHwyDB9uJTgy1Zgx9vu11+zO6NZb43v+sWNh771t9b9YPEE4F2oU03DgclVtrqrNsLIbTyU2LFcl/frZt+9Ro1IdSeWNGWNranTsaKO5nn8epsepZXPLFksQhYUgEnufBg1gzz1tVrtzOSpMgtiiqhMjT1R1EuDjKNPZ0Ufb5K9MnVm9fr2N2or0D1x/vX1gX399fGY9f/IJfPdd7P6HaK1b+x2Ey2lhEsR4ERkSVHI9UkQeBz4QkY4i0jHRAbpKqFbNJs5NmgRz56Y6moqbMAE2bdr2Ab7rrjZ7fNw464Cvqkj/Q3l1qyIJwktxuBwVJkG0B/KAO4A7gVZAB2whob8lLDJXNRdeCLVqweDBqY6k4saMsfIhBQXbtvXrB/vtBzfcYE1EVT1/fn7sGenRWre2qrsrVlTtes5lqDAT5Y4q46dHMoJ0lbD77nD66fDMM7bAUCYZO9bW0KhTZ9u2HXaw4a6zZ8M//1n5c//002+br8riHdUux4UZxbSniAwTkf8Ez1uLyCWJD81VWVGRTfZ64YVURxLe8uUwb17sD/AzzoDOneG226w0emVEymuUNrw1WuvW9tsThMtRYZqYRmC1mPYJni8GrklUQC6Ouna1kUCDBmVOO3qkfyBWgqhWzcqPLF1qNaoqe/6SzVel2WMPqF/fRzK5nBUmQeyuqi8BWwFUdTMQqhFYRIaLyCoRmRu1LV9EpojITBEpFpGDg+27isi/g3kW80Tkokq8HxdNxNruP/kEpk2r+PErVsD551vZiarasMFGV40eXfZ+Y8bY8NK2bWO/ftRR8Lvfwb33Vq7y6pgxtihT7drl7yviI5lcTguTINaLSAOCekwicijwQ8jzjwCOLbFtAHCXquYDtwfPweZXzFfV9kB34AER2SHkdVxp/vAHK/BXkSGvmzfDQw/BgQfCs8/C44/bkqZVMXKkjUK66qrSm4e2boV337Xmn9LmJ4At2bp1qy3L+vPP4WNYutTuBsL0P0S0bm1NXplyB+ZcHIVJENcBrwP7i8hk4Bkg1FdKVZ0AfFdyMxCpG7ErsCJqe10REWDn4Difb1FVdevCuefCiy9abaHyfPSRtfNfey1062Yr3G3dajOaq2LwYKsrVVbz0KxZsHp1+R/geXk2ce6TT2yhpLAf3uWV14ildWubM7F6dfhjnMsWqlruD1ADOAhoA9QMc0zUsc2BuVHPWwFfA0uB5UCzYHtd4H3gG+An4Pjyzt2pUyd1IcycqQqqDz5Y+j7ffqvap4/t16iR6siRqlu32k+LFqo9e1b++rNn23n/9jfV3/1Odddd7Xol3Xef7bdiRbjz3nuv7X/vveH2P+ss1b32svcU1jvv2DXefz/8Mc6lOaBYw3x+h9mpKj8xEsQ/gN7B4zOAd4PHpwEPAgIcAHwB7BLjfH2xCrPFTZs2Tdg/YNY59FDVli23/3DcskV1+HDV3XdXrV5d9brrVH/88bf73HSTvRbrQz2Myy9XrVXLjp89W1XErlPS0Uertm0b/rxbt9qHvojq66+Xve+WLaoNGqief37FYl+61P43efzxih3nXBpL5wTxA9vWoRDgx+Dxm8DhUfuNAw4u69x+B1EBTz9t/7nHjdu2bc4c1YIC237YYaqzZsU+dvp022fYsIpfd9061bp1Vc87b9u2iy5S3WEH1S++2LZt/Xrb9qc/Vez869erduxo15g3r/T9iovtPTz7bMXOv3WrnfvKKyt2nHNpLGyCKLMPQkyTCrRYhbECODJ43INty5d+DRwdXHdPoCWQ4YsapJHTT7chm4MG2WSx66+32cQLFtg61hMn2pDYWDp0sIV1Ro6s+HWff97mYhQVbdt29902ZPW227Zti5TXqEj/AMCOO1r/SJ06VsW2tH6WSHXY8sprlOQjmVwuKy+DAHPCZJpSjn0B61P4FVgGXAIUYAsOzQI+BjoF++4DjAHmAHOBP5R3fr+DqKDrrlOtUUO1SRP7Nn3JJaqrV4c79s9/Vq1ZU/X778Nfb+tW1fx81Xbttm/auvFGi2HGjG2x1apldwSVMWmSxdezp+p//rP9T5cuqu3bV+7cF11kfRfOZQni1cQEPA10CXOyZP94gqigxYvtQ7RtW/tArYgpU+zP5ZlnKn7MoEHbv/b996r166sec4w9b9Omah3hqqpDh9r1Svu5+ebKnXfgQDt+zZqqxedcmgibIMpZwxKAQ4BzReQrYD3Wb6CqWkp7hEtbLVrYymx77ln+8qUldekCjRtbM9N554U7ZtAgm4Nx7rnbv1avnjUxXXstPP20VZ0Ne97SXHKJ1XD6ruTIaqxJq337yp03UpNpwQIb+utcjgjzKdEr4VG45GnUqHLHVasGvXvbfIZ162x+RVm++w7+9S+rKlvavkVF8PDD0LevPa/IBLbS5OVV/RwlRWoyeYJwOSZMNdevgCZAj+Dxz2GOc1notNNsGdM33yx/36eftvIa/fqVvk+tWlYyY9Mmq3tUWid5qjVrZp3g3lHtckyYaq53ADcANwWbagJVqLfsMtZhh8Fee5U/mknV7jS6di2/WefMM62+0jnn2F1KOqpWzZqZPEG4HBOmiekUbIGgGQCqukJEymlfcFmpWjU49VQYMcKWBd1pp9j7vf8+LF5sa1GEOee4cXENMyFat7ZS4c7lkDBf2TYFvd6RYn2lfCq4nNC7txXIe/vt0vcZNMjmXJx+evLiSrTWra2O1Lp1qY7EuaQJkyBeEpEhQD0R6QO8CzyZ2LBc2jriCFutrrRmpo8/tolrF10UrqR2poiMZFq4MLVxOJdEYTqp/waMBEZha1PfrqqVXK3FZbwaNeCUU+CNN6wTOmLtWrj8cut3aNgwPmtIpBNfXc7loLC9gnOAicCE4LHLZb17W7mOMWOsQ/qf/4SWLWHIELj6avuW3axZqqOMr/32s3WxPUG4HBJmFNOlwFTgVKzi6hQRuTjRgbk01qOHre3w6KP2+LzzYN99objYFhraZZfyz5FpatSwJOgJwuWQMKOY+gMdVHUNQLC63IfA8EQG5tJYzZpWGG/ECJsRPXgw9OmTvsNU46V168ot3epchgqTINYA0UM31gXbXC675RZo0gSuvNImueWC1q3hpZdsydQ6dVIdjXMJFyZBfAp8LCKjsaGuJwOzReQ6AFX9ewLjc+nqgAOsbHcuadXK+lwWLbJS6c5luTBtAp8BrxHMgwBGY6u91Q1+nMsNPpLJ5Zhy7yBU9a5kBOJc2mvRAqpX9wThckaW9yo6F0c77GBJwhOEyxGeIJyrCF9+1OUQTxDOVUSrVvDpp1ai3LksV2ofhIg8wraO6e2o6tUJici5dNamDWzZYivgdeyY6micS6iy7iCKgelAbaAjsCT4yQd2SHxozqWhI4+03+++m9o4nEuCUhOEqj6tqk8D7YDuqvpIUKTvaCxJOJd79t4b2ra1OlTOZbkwfRC7AdHFdXYOtjmXm445BiZOtHUxnMtiYRLEfcAnIjJCRJ7GVpa7N7FhOZfGCgutk3rixFRH4lxClZkgRESwBYIOAV4FXgG6Bk1PzuWmww+HWrW8mcllvTJnUquqishbqtoWK7HhnNtxRygo8AThsl6YJqYZItIl4ZE4l0kKC22o64oVqY7EuYQJkyAOAT4Skc9EZLaIzBGR2YkOzLm0Vlhov324q8tiYRJEL2B/oAdwInBC8LtMIjJcRFaJyNyobfkiMkVEZopIsYgcHPVa92D7PBEZX/G34lwStWtna297M5PLYuUmCFX9SlW/An7BZlZHfsozAji2xLYBwF2qmg/cHjxHROoBjwMnqepBwOlh34BzKVGtmg13HTsWtm5NdTTOJUSYNalPEpEl2BoQ44Evgf+Ud5yqTgC+K7mZbXMqdgUiDbjnAK+o6tfBsavCBO9cShUWwqpVMNtbXF12CtPE9H/AocBiVd0Xm0k9pZLXuwYYKCJLgb8BNwXb84DdROQDEZkuIudX8vzOJc8xx9jvsWNTG4dzCRImQfyqqmuAaiJSTVXfBzpX8npFwLWq2gS4FhgWbK8BdAKOx/o8bhORvFgnEJG+Qf9F8erVqysZhnNxsM8+cNBB3g/hslaYBLFWRHYGJgDPicjDwPpKXu8CbLIdwMtApJN6GfCOqq5X1W+Da7WPdQJVfUJVO6tq54YNG1YyDOfipLDQy264rBUmQZwM/Ix9438bW6O63FFMpVgBBOUw6YFVhwWbhFcgIjVEZEdsaO2CSl7DueQpLISNG73shstK5a5JDZwFTFDVJUDoEhsi8gLQHdhdRJYBdwB9gIdFpAawAegLoKoLRORtYDawFRiqqnNjnti5dHLEEbYU6dix0KtXqqNxLq7CJIimwBAR2RdbI2ICMFFVZ5Z1kKqeXcpLnUrZfyAwMEQ8zqUPL7vhsliYeRB3qGoPoDUwEeiPLSTknANrZpozB775JtWRuGwzdy788kvKLh9mHsStIvIfYAxwAPBnoHGiA3MuY0TKbvhwVxcvy5fD6afb4lStW8Obb6YkjDCd1KcCDbCy368Ao1XVvyo5F9G+vZXd8AThqmrzZvj73+HAA+GNN6B/f6hTB044AU49FZYuTWo4YZqYOgI9ganAMcAcEZmU6MCcyxjVqkHPnl52w1XNhx9Cp07wpz/Z4Id582DAAJg5E/76V3j7bWjVCgYOhF9/TUpIYZqY2gDnYnMYzgSWA+MSHJdzmaWwEFautL4I5ypi0ybo0we6dYPvvoNXXrG7h/32s9d32AFuvBHmz4ejj4brr4eOHWFS4r+nh11ydBfgH0ArVT1KVW9PbFjOZZjCQruTeOmlVEfiMs3IkTB0KFxzDSxYAKecAiLb79e8OYwebT8//giPPprw0ES1/MKsIlIHaKqqixIeUQV07txZi4uLUx2Gc+bEE2HaNPj6a/vW51wYF15odwyrVtmXjDDWr4cNG6BBg0pdUkSmq2q5JZPCNDGdCMzEZlFH1nR4vVJROZfNioqsmem111IdicsUqjaHpmfP8MkBYKedKp0cKiJMRHdiNZPWAgQT5PZNYEzOZaZevawZYPDgVEfiMsX8+TZ/JjJUOs2Ereb6Q4ltYRYMci63VK8OffvC++/DwoWpjsZlgsgM/Ejp+DQTJkHME5FzgOoi0kJEHgE+THBczmWmiy+GmjVhyJBUR+IywZgxNuehSZNURxJTmARxFXAQsBF4HvgRW/jHOVfSnnvahKYRI7wEuCvbhg0wfnzaNi9BuIlyP6vqLaraJfi5BdgjCbE5l5mKimDt2soNeV2/Hm66CXr39kl32e7DD63OUqYmCBHpKiKnicgewfN2IvI8MDkp0TmXiY44wma8DhpUseNGj7a6O/fdZ5OlZpZZMNllujFjrDnyyCPL3zdFSk0QIjIQGA70Bt4Ukb9gBfs+BlokJzznMpAI9OsHU6fCjBnl7//ll3DSSfD738Muu8CoUbbdS4hntzFj4LDDYOedUx1Jqcq6gzge6BCs61CI9TscqqoPq+qGpETnXKY6/3wrslbWkNdNm6zGTuvWMG6c1diZMcP6MNq18wSRzVatgk8+SevmJSg7QWyIJAJV/R5YoqpfJiUq5zJdvXpw9tnw3HPwQ8lR4thQ2Pbt4eab4bjjrMTCn/9sTQ5gHxyTJ1ufhMs+771nv9N0eGtEWQliPxF5PfID7FviuXOuLEVFNpLpn//ctm3lSjjvPOjRw9ayfvNNa1IqOcyxsNDuMCZMSG7MLjnGjIH69a3oXhora8nRk0s8fyCRgTiXdTp3tvLNgwZZn8SQIXbH8PPPcOut9rhOndjHFhRA7dr2QXLcccmN2yVWdHmN6tVTHU2ZSk0Qqjo+mYE4l5WKiuDSS62fYfFiu3N4/HFo2bLs4+rUgcMPD98P8dln0LTptiYql77mz4cVK9K+eQnCTZRzzlXWWWdZUbUff4Tnn4d33y0/OUQUFtqHyfLlZe+3ZImd02dvZ4bIyoOeIJzLcTvtZIsILVlindax6vyXJuxa10OGwJYtNhLKpb8xYyyhN2uW6kjKFTpBiMiOiQzEuay1996VG+vetq2V7iirmemXX+Cpp+zxpEnWvu3S18aN8MEHaT+8NSLMehCHich8YGHwvL2IPJ7wyJzLdSLWDFHWWtcjR9oylWedBatX251Kso0eDf/9b/Kvm4kmT7akngHNSxDuDuJBoBewBkBVZwFHJDIo51ygsBC+/RZmzYr9+qBBkJcHt91mzycnuQrOvHk2A/zSS5N73Uw1dizUqAHdu6c6klBCNTGp6tISm7YkIBbnXEk9e9rvWM1Ms2bBRx/ZENpWrawzPAkL2f9GZKb4m29a04krW6S8Rt26qY4klDAJYqmIHAaoiNQUkT8DCxIcl3MOrP+ibdvYCWLwYJsrccEF1hwRHmI6AAAYTElEQVR12GHJTRDr18Mzz1hpkMaN4frrvQ+kLKtXWymVDGlegrInykX0Ax4GGgHLsYJ9VyQyKOdclMJCeOQRm2C3YzBWZN06m6F91lk2Ixdsct2//211fvZIQkX+F16w4bvXXguffgoXXQQvvwxnnJH4a6ez0aPt36aklSvtd4Z0UEO49SC+VdVzVXVPVd1DVf+gqmvCnFxEhovIKhGZG7UtX0SmiMhMESkWkYNLHNNFRDaLyGkVfzvOZaFYZTeeew5++smalyIKCuz3h0lY8FHV+j/atIFu3ax8SNu2Njt806bEXz+d3XUXvPWWlWuP/vnmGzj2WJtdnyHKvYMQkX/E2PwDUKyqo8s5fATwKPBM1LYBwF2q+h8R+V3wvHtwrerA/dhdinMObEZ1rVrWzHTssds+nDt0gIOjvl916mT7TZpkHceJVFxszSWPPmrNW9Wrw/33w+9+Z/MyrroqsddPVz/+aH1Dt90Gd96Z6miqLEwfRG0gH1gS/LQDGgOXiMhDZR2oqhOA70puBnYJHu8KrIh67SpgFLAqRFzO5YaSZTemTIHZs+3uIXriXa1aljCS0Q8xaJBNAjzvvG3bjj0WjjoK7r7bPihz0ZQpNiS5W7dURxIXYRJEO+AoVX1EVR8BegIHAqdg60RU1DXAQBFZCvwNuAlARBoF5yxzGS4R6Rs0TRWvXr26Epd3LgMVFtqQ0uXL7cO5bl0455zt9+vWDaZPT+x62N9/Dy++aNffZZdt20VgwAAbljtgQOKun84mTYJq1eDQQ1MdSVyESRC7AdHTQHcC6qvqFmBjJa5ZBFyrqk2Aa4FhwfaHgBtUtcyFeFX1CVXtrKqdGzZsWInLO5eBIh2b//qXrXV93nmxZ2cXFMDmzTBtWuJieeYZm+xVVLT9a507W8f53/9uBelyzeTJkJ+fMcNYyxMmQQwAZorIUyIyAvgEuwPYCXi3Ete8AHglePwyEGlE7Qy8KCJfAqcBj4tIghtSncsQbdvayKTbbrNyDdGd09EOO8x+J6qZSdWG1x5yiPWBxHLPPZaksqANvkJ+/dWamCKDBbJAmFFMw4DDgNeAV4ECVR2qqutVtX8lrrkCiKzS3QPr10BV91XV5qraHBgJXK6qr1Xi/M5ln2rVbPz8zz9bM1LbtrH32203G1mUqAQxfjwsXFh6ggLYbz+7uxg2zKrRVkQmz6OYOXPbf58sEbZY3wbgG+B74AARCVVqQ0ReAD4CWorIMhG5BOgDPCAis4B7gb4VD9u5HBRpZorVtBOtWzcb6rolAQUPBg2y5VTPPLPs/W691ZrA7rgj/Llfe81mg2dq32IkKWdRgggzzPVS4I/YyKWZwKHYh36P8o5V1bNLeanMgcCqemF553Yu55x9to1UOq2cKUIFBTbUdN48aNcuftdfuRJeeQWuvLL0lfAiGjaEvn3hwQetL2Kffco//8CB1gE+YQL07h2fmJNp8mTYd19o1CjVkcRNmDuIPwJdgK9U9SigA7A2oVE557ZXs6Z9cy9vmcpIG3i8m5mGDbO+hbKal6JddpndxQwdWv6+s2dvm+CX7HpS8aBqcWdR/wOESxAbVHUDgIjUUtWFQMglsZxzSdesmX2LjecH7ZYtdlfSo0f4FfEOOMCaxZ580hJLWQYNsrpS7dsnvyJtPHz2md1hZVHzEoRLEMtEpB7WST1WREYDXyU2LOdcpYnYN9l4Johx4+Drr+2uoCL69YNly6zaa2kidaXOPBNOOMFmaK9fX7V4ky3yb51rdxCqeoqqrlXVO4HbsHkLPvzUuXTWrRssXWof6vEwcqTNnD7xxIodd+KJdjczqIz5r5G6UkVFFveWLfDxx1WLN9kmTbIRZK1apTqSuCozQYhIdRFZGHmuquNV9XVVzfFqXM6lucg32Xg012zZAq++at/uy+ucLqlGDejTB955Bz7/fPvXI3Wl8vOtTEjXrnYHlGn9EJMn2xyUaqFXcc4IZb6bYLb0IhFpmqR4nHPx0LatzeaNxwftxIk29LS80VOlufRS61gfMmT71yJ1pYqKLDHUq2exZ1I/xOrVNjcky5qXIHypjXki8p6IvB75SXRgzrkqqFHDvo3HI0GMHGl3DscdV7njGzWCk06C4cNtFni0WHWlCgpsRFN5HdvpIjL6KkcTxG3ACcDdwANRP865dNatG8yZA2urMCp961ab+3DccdYHUVn9+lkRv1Gjtm1bsyZ2Xalu3axPYs6cyl8vmSZNgh12sDpUWSZMJ/V44EugZvB4GjAjwXE556qqoMDa+KdMqfw5PvrIFrqpbPNSRM+esP/+29awBhgxInZdqUTN40iUSZMsOdSunepI4q7cBCEifbDaSJEGxEbYkFfnXDo75BBr+584sfLnGDnSvh0ff3zVYqlWzRLBxIkwd67dmQweHLuuVNOm0KRJZvRD/PKLlVfPwuYlCNfEdAXQDfgRQFWXAElY8NY5VyU77QQHHWQrnFWGqjUJ9er123UfKuvCC61UyODBNq/i009LrytVUGDJJN2L902bZlVcczhBbIwe1ioiNbBV4Zxz6a5lS1i8uHLHTptmcyniVRdp993h9NNtPYkHHrDCfKWdu6DAajh9leZzciPNYJEy61kmTIIYLyI3A3VE5BhsDYd/JzYs51xc5OXZ/INff634sSNH2miok06KXzxFRTZz+u234eKLS2+3j5SsSPd+iEmTbHJcgwapjiQhwiSIG4HVwBzgMuAt4NZEBuWci5O8PJvo9sUXFTtO1RJEz542Qzheunbd1ufQt4xK/23aWLNWOvdDbN1qQ1yztHkJQpT7xspqPKOqTyY6GOdcnOXl2e/Fi7c9DmPmTEsqN98c33hE4JFH7PwHHFD6ftWrW7NNOt9BzJsHP/yQ1QkizB3EicBiEXlWRE4I+iCcc5kgOkFUxMiR9iH9+wSUXTvySPjjH8vfr6DARjx9/338Y4iHLC3QFy3MPIiLgAOwvoezgc9EJESBd+dcytWvb+3jFUkQkeal7t2tYzlVIh+8kZnK6WbSJNhrL1skKEuFqiylqr8C/wFeBKbj1Vydyxx5eRVLEPPm2f6pXtWtSxfrJE/HfghVW/muWzdrNstSYSbKHSciI4AlQG9gKLBXguNyzsVLRYe6jhxpH3qnnJK4mMLYcUfo1Ck9+yGmT7d1Lqo6gTDNhbmDOB+bOd1SVS9U1bdUNUOqaDnnyMuD5cutvlEYo0ZZ885eafA9sKAApk7dvshfqkWGAJ98cqojSagwfRBnq+prqroRQEQKROSxxIfmnIuLSEf1p5+Wv+/ChdYxXNXaS/FSUGDJYfr0VEeyTaSP5qijrI8ni4XqgxCRDiIyUES+BP4PWFjOIc65dFGRkUyRpUFT3bwUkY4T5mbPtjWo0yWJJlCpQ1ZFJA8btXQ28C3wL0BU9agkxeaci4fIfIMwCWLqVGjWzIrlpYOGDS3BpVNH9ciRVnwwEUOA00xZdxALgR7ACapaoKqPAFuSE5ZzLm7q1LEKqWESRHGxjR5KJwUFliC2bk11JGbUKDjiCNgj+2uWlpUgTgW+Ad4XkSdF5Ggge8dzOZfNwgx1/e47q9uUbgvfFBTY4kIL06Ble/58WLAgJ5qXoIwEEXRMnwUcCLwPXAPsISKDRKQwWQE65+IgLw8WLSq7fHZxsf1OtwTRo4fN6r7hBqsrlUrpMgQ4ScKMYlqvqs+r6olAY+AT4IaER+aci5+8PFt6dM2a0veJJIhOnZITU1jNmsHDD8Mbb8Btt6U2llGjrEbUPvukNo4kCTWKKUJVv1fVJ1T16PL2FZHhIrJKROZGbcsXkSkiMlNEikXk4GD7uSIyW0TmiMiHItK+4m/FOVeqMCOZiouhRQuoVy85MVXE5Zdb9de//hVefDE1MSxebCOYcqR5CSqYICpoBHBsiW0DgLtUNR+4PXgO8AVwpKq2xYbRPpHAuJzLPWETRLo1L0VEqsAefritIzFjRvJjGDXKfp96avKvnSIJSxCqOgH4ruRmILJ24a7AimDfD1U1UrJxCtaU5ZyLl2bNoGbN0hPEypW2ely6JgiwtbFHjrShryefbDEn06hRcPDBNiIsRyTyDiKWa4CBIrIU+BtwU4x9LsEKA8YkIn2D5qni1atXJyhM57JMjRqw//6lJ4hI/0O6DXEtaY89YPRo60s59dTkleD44gubzZ1DzUuQ/ARRBFyrqk2Aa4Fh0S+KyFFYgii1EzzoA+msqp0bNmyY0GCdyyplDXUtLrZmnA4dkhtTZeTnw4gRVgb8iivKHpkVL5HmpVRXuE2yZCeIC4BXgscvAwdHXhCRdlil2JNVtYyhFs65SsnLgyVLYk84mzbN1lbeeefkx1UZZ5wBt9wCw4ZZsghr5Eg7dnMF642OGmXJc7/9KnZchkt2glgBHBk87oGVEEdEmmKJ4zxVreDSV865UPLyYMMGK1MdTTW9O6hLc/fd1iQ2YEC4uwhVGyb78ssVSyrLlsGUKTnXvAQJTBAi8gLwEdBSRJaJyCVAH+ABEZkF3AtEVi2/HWgAPB4ZApuouJzLWZGRTIsW/Xb78uXW4Zvu/Q8lVatmTUwLF8L48eXvP3687Vu3LtxxB6xfH+46rwSNHp4g4icoE763qtZU1caqOkxVJ6lqJ1Vtr6qHqOr0YN9LVXU3Vc0PfjLsq4xzGaC0oa7TptnvTLuDAGsu2m03GDSo/H0HD7Y5HqNGwYoV8NBD4a4xciS0abPt3y+HJLuJyTmXKnvtZX0MJRNEcbGNcmqfgfNT69SBCy+0b/llDXtdudL2ufBCOOYYGyZ7//1Q3kjI6dOt1HiOdU5HeIJwLleIxB7JVFxs35Dr1ElNXFV12WXW6TxsWOn7DB8Ov/4K/frZ8/vug59/hv/7v9KP+e9/LZE0aWJNWTnIE4RzuaRkgsjUDupoLVtaQb8nnohdzG/LFhgyxFaAa9nSth14IFxyiTU7ffbZ9sds3GjzLL7/3uZd5OiQek8QzuWSvDz48sttE8y++MLKfGdyggAoKoKvvoK3397+tXfesdeKin67/c47bXb5Lbf8druq1X766CMb7ZSfn6io054nCOdySV6ezYP4/HN7nikzqMtz8snWxxKrs3rQINhzT9sn2t57w5/+BP/617aOerCaT8OHw623wumnJzbuNOcJwrlcUnIk07RpVuOoTZvUxRQPNWvCpZfCW2/Z3ULEV1/ZOtuXXmrvs6T+/a356Prr7c7hvffguussmdx1V/LiT1OeIJzLJS1a2O9IgiguttFLsT48M02fPtYR/0RUMegnn9z2WiyROREffACPPmp3DAceCM8+a/Mscpz/CziXS+rVs4J3ixdbU9P06Znf/xDRtCkcf7yNZtq0yX6GDrVtzZqVflzfvpY4r77aEszo0ZY4nCcI53JOZCTT4sWwbl3m9z9EKyqyOQ+vvWYf9CtXbt85XVLNmvD3v0P9+vDSS1b11gFQI9UBOOeSLC/P2urTdQ3qqigshObNrWNaxO4cevUq/7gTTrBJc96s9Bv+r+FcrsnLs0lg48bZ5LhWrVIdUfxUr24T5z74AN5/3x5Xrx7uWE8O2/F/EedyTWQk0yuvQMeOVmYjm1x8sTUb1axpj12lZdlfhnOuXJEE8cMP2dW8FLHHHnDDDTZsdc89Ux1NRvME4Vyu2X9/a59Xzc4EAWXXWHKheROTc7mmdu1twz6zNUG4uPAE4Vwuysuzsf45uMaBC8+bmJzLRf37WxkKH7njyuAJwrlc1LNnqiNwGcC/PjjnnIvJE4RzzrmYPEE455yLyROEc865mDxBOOeci8kThHPOuZg8QTjnnIvJE4RzzrmYRFVTHUOlichq4Ktydyzd7sC3cQonk/j7zi3+vnNLmPfdTFUblneijE4QVSUixaqac9XK/H3nFn/fuSWe79ubmJxzzsXkCcI551xMuZ4gnkh1ACni7zu3+PvOLXF73zndB+Gcc650uX4H4ZxzrhQ5mSBE5FgRWSQin4rIjamOJ1FEZLiIrBKRuVHb6ovIWBFZEvzeLZUxJoKINBGR90VkvojME5E/Btuz+r2LSG0RmSois4L3fVewfV8R+Tj4e/+XiOyQ6lgTQUSqi8gnIvJG8DxX3veXIjJHRGaKSHGwLS5/6zmXIESkOvAYcBzQGjhbRFqnNqqEGQEcW2LbjcB7qtoCeC94nm02A39S1dbAocAVwX/jbH/vG4EeqtoeyAeOFZFDgfuBB1X1AOB74JIUxphIfwQWRD3PlfcNcJSq5kcNb43L33rOJQjgYOBTVf1cVTcBLwInpzimhFDVCcB3JTafDDwdPH4a+H1Sg0oCVf1GVWcEj9dhHxqNyPL3ruan4GnN4EeBHsDIYHvWvW8AEWkMHA8MDZ4LOfC+yxCXv/VcTBCNgKVRz5cF23LFnqr6TfD4v8CeqQwm0USkOdAB+JgceO9BM8tMYBUwFvgMWKuqm4NdsvXv/SHgemBr8LwBufG+wb4EjBGR6SLSN9gWl791X5M6h6mqikjWDmMTkZ2BUcA1qvqjfak02freVXULkC8i9YBXgQNTHFLCicgJwCpVnS4i3VMdTwoUqOpyEdkDGCsiC6NfrMrfei7eQSwHmkQ9bxxsyxUrRWRvgOD3qhTHkxAiUhNLDs+p6ivB5px47wCquhZ4H+gK1BORyJfBbPx77wacJCJfYk3GPYCHyf73DYCqLg9+r8K+FBxMnP7WczFBTANaBCMcdgDOAl5PcUzJ9DpwQfD4AmB0CmNJiKD9eRiwQFX/HvVSVr93EWkY3DkgInWAY7D+l/eB04Ldsu59q+pNqtpYVZtj/z+PU9VzyfL3DSAiO4lI3chjoBCYS5z+1nNyopyI/A5rs6wODFfVe1IcUkKIyAtAd6y640rgDuA14CWgKVYJ9wxVLdmRndFEpACYCMxhW5v0zVg/RNa+dxFph3VIVse+/L2kqneLyH7YN+v6wCfAH1R1Y+oiTZygienPqnpCLrzv4D2+GjytATyvqveISAPi8LeekwnCOedc+XKxick551wIniCcc87F5AnCOedcTJ4gnHPOxeQJwjnnXEyeIFxaExEVkQeinv9ZRO6M07lHiMhp5e9Z5eucLiILROT9OJxraFWLS4pI8+gKv86VxhOES3cbgVNFZPdUBxItaoZuGJcAfVT1qKpeV1UvVdX5VT2Pc2F4gnDpbjO2hOK1JV8oeQcgIj8Fv7uLyHgRGS0in4vIfSJybrBWwhwR2T/qND1FpFhEFgc1fSIF7waKyDQRmS0il0Wdd6KIvA5s9yEtImcH558rIvcH224HCoBhIjIwxjH9o64TWb+huYgsFJHngjuPkSKyY/DaByLSOYhxRHCtOSJybfB6vohMCc73amQdABHpJLZOxCzgiqjrx3yvzoEnCJcZHgPOFZFdK3BMe6Af0Ao4D8hT1YOxctBXRe3XHKtdczwwWERqY9/4f1DVLkAXoI+I7Bvs3xH4o6rmRV9MRPbB1h/oga3F0EVEfq+qdwPFwLmq2r/EMYVAi+D6+UAnETkieLkl8LiqtgJ+BC4v8f7ygUaq2kZV2wJPBdufAW5Q1XbYTPI7gu1PAVcFa0VEK+u9uhznCcKlPVX9Efvgu7oCh00L1oXYiJW8HhNsn4MlhYiXVHWrqi4BPseqnxYC5wdlsz/GSke3CPafqqpfxLheF+ADVV0dlJh+Djgixn7RCoOfT4AZwbUj11mqqpODx//E7kKifQ7sJyKPiMixwI9BAq2nquODfZ4GjgjqM9UL1gcBeLZEDKW9V5fjvNy3yxQPYR+iT0Vt20zwJUdEqgHRS0pG19zZGvV8K7/9uy9Za0YBwb5tvxP9QlDnZ33lwo9JgL+q6pAS12leSlzbnqh+LyLtgV7YndIZxGiGCxnDdu/VOfA7CJchgkJjL/HbZSO/BDoFj0/CVlCrqNNFpFrQL7EfsAh4BygKSoYjInlBpcyyTAWOFJHdxZa1PRsYX84x7wAXi61bgYg0EqvpD9BURLoGj88BJkUfGHTaV1PVUcCtQEdV/QH4XkQOD3Y7DxgflP5eGxQxBDi3RAwVfa8uR/gdhMskDwBXRj1/EhgddLy+TeW+3X+NfbjvAvRT1Q0iMhRrhpohIgKsppwlG1X1GxG5ESsxLcCbqlpmiWVVHSMirYCP7DL8BPwB2IIlqitEZDjWIT6oxOGNgKeCOyeAm4LfF2B9KTtizVAXBdsvAoaLLRwzJuo8FX6vLnd4NVfn0kzQxPSGqrZJcSgux3kTk3POuZj8DsI551xMfgfhnHMuJk8QzjnnYvIE4ZxzLiZPEM4552LyBOGccy4mTxDOOedi+n9QEoeAtJqc2gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xm81nP6+PHX1UKLiBZGoaJyKq2nlE4py7EUIVuEIXEaM5bf0DdLNGYylmaYsZQSMmaMJDKN5YSiQnVaKEVJUZaRlIi0Xb8/rs9dp9M59/05y33u7Xo+Hvfjvu/P/fl87uvOcV/3571cb1FVnHPOOYAqiQ7AOedc8vCk4JxzbhdPCs4553bxpOCcc24XTwrOOed28aTgnHNuF08KzjnndvGk4JxzbhdPCs4553aplugASqt+/frapEmTRIfhnHMpZf78+d+qaoNY+6VcUmjSpAkFBQWJDsM551KKiHwWZj9vPnLOObeLJwXnnHO7eFJwzjm3S8r1KTjnKs+2bdtYu3YtW7ZsSXQoLqQaNWrQuHFjqlevXqbjPSk450q0du1a6tSpQ5MmTRCRRIfjYlBV1q9fz9q1a2natGmZzuHNR865Em3ZsoV69ep5QkgRIkK9evXKdWXnScE5F5UnhNRS3v9eGdN8tGQJTJxYumNE4IILoFWr+MTknHPJJmOSwrJl8Kc/le4YVRgzBgoK4LDD4hOXcy66r7/+muuvv5558+ZRt25dDj74YB544AH22Wcf+vbty5IlSygoKOCpp57i73//e9zimDFjBqNGjWLq1Kl7be/Xrx9NmzZly5Yt9O3bl1GjRkU916JFi/jyyy85/fTT4xZvWWVM89F558HOnaW7LV0KP/8MZ50FP/2U6E/gXOZRVc4++2x69erFypUrmT9/Pn/+85/53//+t8d+2dnZcU0IsfTo0YNFixaxcOFCpk6dyuzZs6Puv2jRIl5++eVSvYeqsnPnzvKEGUrGJIWyyMqCf/0LFi6EQYPsysE5V3mmT59O9erVycvL27WtXbt29OjRY4/9ZsyYQd++fQEYMWIEl1xyCd26daN58+aMGzdu1z49e/akT58+tGzZkry8vF1fsvn5+XTr1o2OHTty3nnn8eOPPwLw6quvcvTRR9OxY0cmT54cM96aNWvSvn17vvjiCwA2b97MFVdcQZcuXejQoQNTpkxh69at3H777Tz77LO0b9+eZ599lhEjRuxxddGmTRtWr17N6tWradmyJZdeeilt2rRhzZo17Lffftx66620a9eOrl277pUgyytjmo/Kqm9fGDkSbrkF2rWDYcMSHZFziXH99bBoUcWes317eOCBkl9fsmQJnTp1KvV5P/jgA9577z02b95Mhw4d6NOnDwBz585l6dKlHHHEEZx66qlMnjyZXr168ac//YnXX3+d2rVrc8899/DXv/6VoUOHMnjwYN58802OOuooLrjggpjvu2HDBlasWEHPnj0BGDlyJCeccAKPP/44GzdupEuXLpx00knceeedFBQU8NBDDwGWyEqyYsUKJkyYQNeuXQFLNF27dmXkyJEMHTqUcePGcdttt5X636gknhRCGDYMPvjAEkObNpYonHPJq1+/ftSsWZOaNWvSu3dv5s6dS926denSpQvNmjUDYMCAAcyaNYsaNWqwdOlSunfvDsDWrVvp1q0bH330EU2bNqV58+YADBw4kLFjxxb7fjNnzqRdu3asWLGC66+/nkMOOQSwK5CXXnpp11XAli1b+Pzzz0v1WY444ohdCQHY1ZcC0KlTJ6ZNm1aq88XiSSEEERg/HpYvh4sugvfe8xFJLvNE+0UfL61bt2bSpEmlPq7osMzI8+K2qyonn3wyzzzzzB6vLSrFZVGPHj2YOnUqq1atomvXrpx//vm0b98eVeX555+nZcuWe+w/Z86cPZ5Xq1Ztj/6CwvMMateuvce+1atX3/U5qlatyvbt20PHGYb3KYRUqxa8+CLUrAn9+sGGDYmOyLn0d8IJJ/DLL7/s8Qv9gw8+YObMmVGPmzJlClu2bGH9+vXMmDGDzp07A9Z8tGrVKnbu3Mmzzz5LTk4OXbt2Zfbs2XzyySeANc8sX76co48+mtWrV7Ny5UqAvZJGcZo2bcqwYcO45557ADjllFN48MEH0aBDcuHChQDUqVOHH374YddxTZo0YcGCBQAsWLCAVatWhfr3iQdPCqVw2GEweTJ89hkMGZLoaJxLfyLCCy+8wOuvv86RRx5J69atufnmm3c1z5Skbdu29O7dm65duzJ8+HAOPfRQADp37sxvf/tbsrKyaNq0KWeffTYNGjTgySefZMCAAbRt23ZX01GNGjUYO3Ysffr0oWPHjjRs2DBUzHl5ebz99tusXr2a4cOHs23bNtq2bUvr1q0ZPnw4AL1792bp0qW7Opr79+/Pd999R+vWrXnooYdo0aJF+f7hykE05JAaEamlqqUamCki1wGDAQHGqeoDRV4/GngC6AjcqqrRB/cC2dnZmuhFdgYPhuefh/XrrWnJuXS1bNkysrKyEh1GqYwYMYL99tuPG2+8cY/tJc0zSEfF/XcTkfmqmh3r2JhXCiJynIgsBT4KnrcTkUdCHNcGSwhdgHZAXxE5qshu3wHXAjGTQTLp0MGaj9auTXQkzjlXscJ0NN8PnAK8BKCq74tIzxDHZQFzIlcXIvIWcA5wb2QHVf0G+EZE+pQ28ERq187uP/jAZzo7l2xKGt7Zq1cvevXqVamxpKJQfQqquqbIph0hDlsC9BCReiJSCzgdSIuv0GOOsfv3309sHM5VhrBNzC45lPe/V5iksEZEjgNURKqLyI3AshCBLQPuAfKBV4FFhEsmexGRq0SkQEQK1q1bV5ZTVKj994emTe1Kwbl0VqNGDdavX++JIUVE1lOoUaNGmc8RpvkoD/gb0Aj4AvuSvyZkgOOB8QAichdQplZ4VR0LjAXraC7LOSpa27Z+peDSX+PGjVm7di3J8GPMhRNZea2sYiYFVf0WuLgsJxeRhqr6jYgcjvUndI11TKpo1w7+8x8rmFezZqKjcS4+qlevXuYVvFxqKjEpiMiDQIm/ylX12hDnf15E6gHbgGtUdaOI5AXHjxGRQ4ACYH9gp4hcD7RS1U2l+RCJ0LatVVL98EPIjjnIyznnUkO0K4XIZIDuQCvg2eD5ecDSMCdX1R7FbBtT6PHXQNmvcxKo8AgkTwrOuXRRYlJQ1QkAIjIEyFHV7cHzMUD0OeYZoFkzqF3b+xWcc+klzOijA7HmnYj9gm0ZrUoVG5rqI5Ccc+kkzOiju4GFIjIdK1fRExgRz6BSRdu28NxztviOl7twzqWDmFcKqvoEcCzwAvA80C3StJTp2rWzchfBIkvOOZfywlZJ7QL0wK4SOscvnNTStq3de7+Ccy5dhCmIdzdwHTbiaClwbTARLeNFyl14v4JzLl2E6VM4HWivqjsBRGQCsBC4JZ6BpYIDDoAmTfxKwTmXPsI2H9Ut9PiAeASSqtq18ysF51z6CHOl8Gf2Hn00LK5RpZC2bb3chXMufYQZffQMVrNoMrtHHz0b/ajM0a6dlbtYGmqOt3POJbcwHc3dgU2q+hI2iW2oiBwR98hShI9Acs6lkzB9CqOBn0SkHfD/gJXAU3GNKoUceSTUquX9Cs659BAmKWxXW2GjH/Cwqj4M1IlvWKkjUu7CrxScc+kgTFL4QURuBgYC/xWRKkD1+IaVWiIjkHxxKudcqguTFC4AfgEGFSp1fV9co0oxbdvCd995uQvnXOoLs/La18BfCz3/HO9T2EPhtRXKsQqec84lXIlXCiIyK7j/QUQ2Fb2vvBCTX6TchfcrOOdSXbRFdnKCe+9UjiFS7sJHIDnnUl2YGc2ISEcgB1uzeZaqLoxrVCmobVtPCs651Bdm8trtwASgHlAfeFJEbot3YKmmXTv4+GPYsiXRkTjnXNmFuVK4GGinqltgVyntRcCf4hlYqmnbFnbssHIXHTsmOhrnnCubMENSvwRqFHq+L+CDL4uIjEDyzmbnXCoLc6XwPfChiEzD+hROBuaKyN8BVPXaOMaXMpo183IXzrnUFyYpvBDcImbEJ5TUVrWql7twzqW+MJPXJohITeBwVf24EmJKWW3bwnPPwZo1cNhh8XmPKVPg00/hhhvic37nXGYLM/roDKxj+dXgeXsReSnegaWiAQPgl18gKwtGjYJt2yr2/Bs3whVXwI03WuJxzrmKFqajeQTQBdgIoKqLgGZxjCll9e4NH35o9zfdZKOQZs2quPPffTds2GCF98aNq7jzOudcRJiksE1Vvy+ybWc8gkkHTZva8pwvvgibNkGPHvbrft268p13zRr4299g4EA47TR47LGKvxJxzrkwSeFDEbkIqCoizUXkQeCdOMeV8vr1szkL//d/8I9/WBmMxo33vjVtCs+GWNz09ttt2c8//hGGDIGvvoKXvBHPOVfBRGMsAiAitYBbgdxg02vAnyKT2WIcex0wGBBgnKo+UOR1Af4GnA78BPxaVRdEO2d2drYWFBTEeuuksnQpPPJI8bOd58+HZcvg7behS5fij1+82OZB/P73cN99NkmuWTNo3hxefz2+sTvn0oOIzFfV7Jj7xUoK5QigDfBvrD9iK9ZRnaeqnxTa53Tgd1hSOBb4m6oeG+28qZgUovn2W+jc2TqoCwrg0EP33uf00+Hdd2HlSjjoINs2ciTcdpuV1mjRonJjds6lnrBJIUzzUVllAXNU9SdV3Q68BZxTZJ9+wFNq3gPqisiv4hhT0qlf35qBNm2Cs8/e+2rizTfhlVfg1lt3JwSAQYOgWjV49NHKjTcd7PQeMedKFM+ksAToISL1giao04Gio/cbAYUHV64NtmWUY46xfoe5c+Hqq3cv67lzJwwdCocfDr/97Z7HHHKIJZEnn4Sff670kFOWKnTqBB062NWXc25PUZOCiFQVkTJNk1LVZcA9QD7WdLQI2FGWc4nIVSJSICIF68o7jCdJnX02jBgBTz0F999v2yZOtD6HP/4RatTY+5ghQ2wZ0Oeeq9RQU9qKFbBokfXzHHccXHWV/Rs650zUpKCqO4ABZT25qo5X1U6q2hPYACwvsssX7Hn10Jhiiu2p6lhVzVbV7AYNGpQ1nKQ3fDj0729zHP7zH7jlFutgvvji4vfv1QtatoQxYyo1zJSWn2/3c+dax/3jj9u/4RNPeLOScxCu+Wi2iDwkIj1EpGPkFubkItIwuD8c60/4V5FdXgIuFdMV+F5VvyrNB0gnVapYc1CbNjakddUquOceq6tUHBHIy7NmkHSpufT11zbcNl6mTbORW+3a2azzhQstKVxxBRx/vI0Ecy6ThRmSOr2YzaqqJ8Q8uchMbHGebcD/U9U3RCQvOMGYYEjqQ8Cp2JDUy1U16tCidBt9VJzVq214aseO1sksUvK+GzbYiKXLLkuPK4YePSwpLF0K++xTsefets066wcOhNGjd2/fuRMmTLDyIYcfbonCuXST8CGp8ZIJSQGszlHNmrDvvrH3vfxymDQJvvwS6qTwitobNthorJ07bfb2tRVclH3mTOjZEyZPtj6com69Fe69FzZvrviE5FyiVdiQVBE5WETGi8grwfNWIjKoIoJ0JatbN1xCAOtw/vFHePrp+MYUb9OnW0I47DC48074vmhxlXLKz7cmut69i389Kwu2b4dPPin+decyQZg+hSexWcyRaVXLgevjFZArvc6dbYjl6NG7h7Omovx8u9J57jlYv95+tVekadPg2GMt4RanVSu7X7q0Yt/XuVQSJinUV9WJBEXwgoloZRpa6uJDxK4WFi9O7bH3+fn2K/7YY+Gii2xo7hcVtPDrd9/BvHmQm1vyPkcfbf+W3tnsMlmYpLBZROphS3ESGSUU16hcqQ0YAPvvn7qdzStX2miryJf2n/5kNZ7uuKNizv/mm9Y0FS0p1KplhQv9SsFlsjBJ4f9hQ0ePFJHZwFNYvSKXRPbbDy65xCa8rV+f6GhKLzJ/IPKl3bQpXHONzR/48MPyn3/aNEuaJRUdjGjVypOCy2wxk0JQtfR44DjgaqC1qvry9EkoL88K6z3xRKIjKb38fPuVftRRu7fdeqv1MQwbVr5zq8Jrr8EJJ1i9qGhatbIig9u3l+89nUtVYUYf1QCuBf4I/AG4JtjmkkybNpCTY0XyUml27vbt1rxz8sl7zsmoV88SwtSp8NZbZT//J5/AZ59FbzqKyMqyxLpqVdnfz7lUFqb56CmgNfAgNtGsNfCPeAblym7IEPsSfOONREcS3ty5ViW2uC/t666zxYhuuqnsI6siTVMnnxx7Xx+B5DJdmKTQRlUHqer04DYYSwwuCfXvbxPAUqnDOTJ/4IRi5sjXrGlzFubNK3vhv2nTrI/iyCNj75uVZfc+AsllqjBJYUEw4ggAETkWSP8pxSlq332tjs+UKWUbzvnxx3DWWfDnP5c/lm++gW7dYM6c6Pvl50N29p7rRRR26aXWNDZsWOkntG3bZk1TubnRy4VE7L+/XZn4lYLLVGGSQifgHRFZLSKrgXeBziKyWES8wzkJXX21Ded87LHwx/z8s1VpbdvWEsq998LWreWLY+xYeO89+N3vSm762bjRmo+itfdXrWrLma5ZYxVjd5RilsycOfDDD+H6EyJ8BJLLZGGSwqlAU2wE0vHB41OBvsAZ8QvNlVWzZnDKKTBuXLhRNC+/DK1b29yA88+34zZutLITZbVjhyWFAw+M3vQzfbrtG+tLu0cPq4f03//aqKSwojVNlaRVK2s+SqXOeucqSpghqZ9Fu1VGkK70hgyx5qOpU0veZ80aOOcc6NPHmp3efNNWgBs40OY9TJpU9vd/+WU7/5gx1vRzyy3FX3nk59t7de2692vFfaarrrJy4v8qWoS9BNOm2dyEkkpbFCcrC376CT7/PPwxzqWLeC7H6RKoTx9rGy+uw3nbNltLICsLXn0V7rrL1mOIFIqrUQPOOANefLHs4/XHjIFf/cqqkd5zj81YLm496WnT7H2rV499ThF48EG7ahg0CGIVy92wIXbTVHEiI5C8s9llIk8KaapaNRg82CZtrVy5e/vs2bZOw003WZPK0qVw8817l4o+91z49lt4++3Sv/eqVbYOxJVX2pf9aafZF/+dd9rQ04iVK+1Wmi/tffaxK5iGDa1D/OuvS943UtoizFDUwiIjkLxfwWUiTwpp7MorrZP20UftC/6KK2xy26ZN1pn80ks2i7g4p55qtYDK0oQ0bpz9qh882J6LWMf1t9/Cffft3m/aNLsv7Zd2w4YW/4YN1vz1yy/F7xepunrssaU7f716cPDBnhRcZipxkR0R+YGgCF5xVHX/eAUVTaYsslNR+ve3iWxVq1oy+P3vbZRR7dqxjz3vPJg1C9auLXlJ0KK2brX1ELp2tS/uwgYMsG2ffGKrxfXvb01Aq1eHGy5a1HPPWcf4xRdbP0hRV19tJcVffLH05z7hBBuRlcpVZ50rLOwiO6hq1BtW3uI3QB1gf2AIcGes4+J169Spk7rwpk9XBdWePVWXLCndsf/+tx379tulP+aVV/Z+beVK1erVVQcPVt22TfWAA1SvvLJ0MRU1fLi9X0m3sWPLdt5rrlHdf3/VnTvLF59zyQIo0BDfsTHKgwFwpqq2K/R8tIi8D9xe2kzlKl+vXlb357DDSv9r/PTTbVTSpEnWuRvG6NE2e7i4foJmzeA3v7HO4q5dbSJaaZuOivrDH+yKZvPmvV+rXh3aty/bebOy7Mrqyy+hUaPyxehcKgmTFDaLyMXAv7HmpAFAMf8LumR1+OFlO65OHetbmDzZFrypEqMHaulSK1x3990l73vbbVbFdcgQS1Innli22CJE4JhjyneO4hQegeRJwWWSMB3NFwHnA/8LbucF21wGOPdc61OYOzf2vo8+ar/OL7+85H3q17dyFVu3WmmLevUqLtaK5IXxXKaKmhREpCpwtqr2U9X6qtpAVc9S1dWVE55LtL597Ys+1iikzZthwgRLIg0bRt/3uuvsS3fAgIqLs6I1bGi1mDwpuEwTNSmo6g6suchlqLp1rd3/+eejl65+9lnrIxgyJPY5a9Wy1dRuuKHi4qxoIl4DyWWmMM1Hs0XkIRHpISIdI7e4R+aSRv/+Nmx0wYKS9xk92uon5eRUWlhx50nBZaIwSaE9tn7CncBfgtuoeAblkku/fjZPoaQmpBdesPkGeXllm2+QrLKybL3rdesSHYlzlSdMQbzexdxKUXPSpbp69Wwy16RJezYhffEFXHCBzSrOyoJLLklcjPHgnc0uE4UqcyEifURkqIjcHrnFOzCXXPr3t5nIixdbkbwHHrBEMGWK1TRauBAOOCDRUVYsTwouE8VMCiIyBrgA+B0g2JDUI+Icl0syZ51lcw9GjrShpDfcAN27W4fx8OE2yS3dNGpkczU8KbhMEuZK4ThVvRTYoKp/ALoBLeIblks2Bx8MPXvCxIlW2G7SJFszIcy6x6nKRyC5TBQmKfwc3P8kIocC24BfhTm5iNwgIh+KyBIReUZEahR5/QgReUNEPhCRGSLSuHThu8p0zz229sKyZdaclE6dyiXxpOAyTZikMFVE6gL3AQuA1UDMda9EpBFwLZCtqm2AqsCFRXYbBTylqm2x0U0VsFy8i5cuXWzthTp1Eh1J5cnKsjUbNmxIdCTOVY4wo4/+qKobVfV5rC/haFUN29FcDagpItWAWsCXRV5vBbwZPJ4O9At5Xucqha/C5jJNmI7mWSIyUkROBfZR1e/DnFhVv8CuBD4HvgK+V9X8Iru9D5wTPD4bqCMiSVoNx2UiH4HkMk2Y5qNLgI+B/sA7IlIgIvfHOkhEDsR++TcFDgVqi0jRpVBuBI4XkYXA8cAXwI5iznVV8L4F63wmkatERxwBNWt6UnCZI2bpbFVdJSJbgK3BrTeQFeLcJwGrVHUdgIhMBo4Dni507i8JrhREZD+gv6puLCaGscBYsJXXQry3cxWiShXrV/Ck4DJFmOajlcCLwMHAeKCNqp4a4tyfA11FpJaICHAisEfLrIjUF5FIDDcDj5cmeOcqQ1aW9ym4zBGm+ejv2Bf8AGw00WUiEnN0uqrOASZhI5YWB+81VkTuFJEzg916AR+LyHIs6Yws9SdwLs7atIHPP7c6SM6lO9Fo9ZAL72jNO5dj/QCNVTXkUu4VKzs7WwsKChLx1i5DvfcedOtm5cHPPz/R0ThXNiIyX1WzY+0XpvnoLyIyB5gDtMXWZm5e/hCdSw3Z2bauRH7RsXPOpaEwazS/C9yrqv+LdzDOJaNq1axKbH6+VYnNhJncLnOF6VOYDJwsIsMBRORwEekS37CcSy65ubBmDSxfnuhInIuvMEnhYawI3kXB8x+Cbc5ljNxcu/cmJJfuwiSFY1X1GmALgKpuAPaJa1TOJZmmTa0irCcFl+7CJIVtIlIVUAARaQDsjGtUziWh3FyYPh22bk10JM7FT9h5Ci8ADUVkJDALuCuuUTmXhHJzYfNmG6LqXLoKU+binyIyH5uRLMBZqurzO13G6d0bqla1JqSePRMdjXPxEWqNZlX9SFUfVtWHgK9E5NY4x+Vc0jngADj2WO9XcOmtxKQgIoeJyFgRmSoiV4pIbRH5C7AcaFh5ITqXPHJzoaDAS1649BXtSuEpbFGcB4HWQAFWArutql5XCbE5l3Ryc20C25tvxt7XuVQULSkcpKojVPU1Vb0BqANcrKpfV1JsziWdzp2tGcmbkFy6itrRHCyUE5nUvx44ICiDjap+F+fYnEs6XvLCpbtoSeEAYD67kwJYGWywOQvN4hWUc8ksNxdeeMFKXrRsmehonKtYJSYFVW1SiXE4lzJOPtnup03zpODST6ghqc653Y48Epo1834Fl548KThXBl7ywsWDKixcCNu3Jy4GTwrOlUFuLvz4o5e8cBXno4/gxBOhY0db2OnddxMTR7TJawdFu1VmkM4lm969oUoV61dwrjx++gluvRXatrWrhFtuscmRxx0HgwdX/kTJaFcK87EJa/OBddhM5hXB4/nxD8255FW3rpe8cOU3dSq0bg133QUDBsDHH8PIkbBsGfz+9/DEE3D00Xa/s5JqU5eYFFS1qao2A14HzlDV+qpaD+gL+P8KLuPl5sK8efCdz9hxpbRxI5x9NpxxBtSqBTNmwIQJ0DAoILTffjBqlF05tGwJV1wBxx8PS5bEP7YwfQpdVfXlyBNVfQU4Ln4hOZcaTj/dOgaffz7RkbhU88gj8OKLcPfd9sV//PHF73fMMfD22/D443b18PTT8Y9NVDX6DiKvATOBSDgXAz1V9ZQ4x1as7OxsLSgoSMRbO7cHVWsHrl4d5s/32c0uvF69YNMmWLAg5q67rF8PNWpA7dple08Rma+q2bH2C3OlMABogC20Mzl4PKBsYTmXPkRgyBD7pTdvXqKjcanixx/hnXd2r/sdVr16ZU8IpREzKajqd0FV1BxV7aiq13vdI+fMwIH2P+qYMYmOxKWKt96CbdtKnxQqS8ykICLHichSYFnwvJ2IPBL3yJxLAfvvDxdfDP/+N2zYkOhoXCrIz4eaNaF790RHUrwwzUf3A6dgVVJR1fcBX4zQuUBeHvz8Mzz1VKIjcakgP986lvfdN9GRFC/scpxrimzaEYdYnEtJHTrYnIUxY6zz2bmSrFljM5eTtekIwiWFNSJyHKAiUl1EbiRoSnLOmSFD7H/2t94q/bHr19vxv/lNxcflkktkBnyqJ4U84BqgEbY8Z/vgeUwicoOIfCgiS0TkGRGpUeT1w0VkuogsFJEPROT00n4A55LB+efDgQfC6NHhj9m508aft2xpVxljxsC6dfGL0SVefj4ceii0apXoSEoWZvTRt6p6saoerKoNVHWgqsasxiEijYBrgWxVbQNUBS4sstttwERV7RC85h3YLiXVrAm//jVMngz/+1/s/T/4AHr0gEGDICsLxo+3pqc33oh7qC5BduywK4WTT07uOS1hRh81E5H/iMg6EflGRKaISNhV16oBNUWkGlALu9IoTIH9g8cHFPO6cynj6qut5PH48SXv88MPVtOmY0erc/P449bkdNlldqXhtZTS18KFVhIlmZuOIFzz0b+AicCvgEOB54BnYh2kql8Ao4DPga/+rHWTAAAcl0lEQVSA71W16J/8CGCgiKwFXgZ+Fzpy55JMy5a2fvOjj9qvwsIi5TCysuCvf7VaNh9/DJdfbtVWq1a1ssnTpnlndbqK9CecdFJi44glTFKopar/UNXtwe1poEasg0TkQKAf0BRLJrVFZGCR3QYAT6pqY+B04B8isldMInKViBSISME6b3R1SWzIEPj8c3j11d3bVq60Oknnngv169ts1rFjbYZqYbm5sHatdVi79JOfD+3b7y56l6zCJIVXRGSYiDQRkSNEZCjwcoh1FU4CVqnqOlXdhpXIKFpIbxB2FYKqvoslm/pFT6SqY1U1W1WzGzRoEOZzOZcQ/frBIYdYh/Mvv8Cdd1pp5Fmz4P77oaAAunUr/tjI2s/ehJR+fvwRZs9O/qYjCJcUzgeuBqYDM4AhWKdwZL2FknwOdBWRWiIiwInsPZT182A7IpKFJQW/FHApq3p1uPJKePllaNMG7rjDEsVHH8H110O1aiUf26QJNG8ePil8/LE3NaWKSGmLSOJPZmFGHzWNciuxw1lV5wCTgAXA4uC9xorInSJyZrDb74HBIvI+1k/xa41VttW5JHfVVZYcAF57DZ59Fho1Cndsbq7V1v/ll+j7zZhhi6/897/lidRVlmnTrMJpTk6iI4ktyu8WE8wt+A2Qg40WmgmMUdUtsY5V1TuAO4psvr3Q60uBJK0A4lzZHHYYfPIJNGhgXwSlkZsLDz9s6/P26lXyfg89ZPdvvgl9+5Y5VFdJIqUtSvv3kAhhmo+eAloDDwIPBY//Ec+gnEt1hx1Wti+AXr1sJFK0JqQvv7QFWsD6KlxyW7PGFshJhf4ECJcU2qjqIFWdHtwGY4nBOVfB9t/fOqKjJYXx423I6/nn2yItmzdXXnxg/RjPPGNzLlxskaGoqdCfAOGSwgIR6Rp5IiLHEr2D2TlXDrm59mX/7bd7v7Z9uw1nzc21GdQ7dsDcuZUb3yuvwEUXwc03V+77pqpp02xEWps2iY4knDBJoRPwjoisFpHVwLtAZxFZLCIfxDU65zJQbm7JJS9eftnmMuTl2RWFSOU3IUXqOz36KKxYUbnvnWp27rSkkJub3KUtCguTFE7FJqAdH9yaBtv6AmfELzTnMlN2NtStW3wT0ujRNpLpjDNsnzZtKjcpfPaZjXgaNMjWA7jllsp771S0cKFVwU2VpiOIMvpIRPZX1U1AsS2HviSnc/ERKXmRn29XDJFfmJ9+akNc77hj93yHnBx4+mlrRqpaNf6xjRtn98OHQ+PG8Ic/wJw5tp5EJhs/fnffQWGffmr3yV7aorBoVwr/Cu4jk9TmF7p5n4JzcRQpefHxx7u3jR1rdZKuvHL3tpwc6/BdvDj+MW3dCo89Bn36wBFHWGG/hg1h6NDMnkS3c6f9G0ybBosW7XnbtMnqWx1ySKKjDK/EKwVV7RvcN628cJxzsGfJi6OPtsls48fDmWfuOREuMhlq1iyrqxNPU6ZYWfC8PHtepw6MGGGLA02dak1ameijj6z66RNPWOd/qivxSkFEOka7VWaQzmWapk3hqKN29ytMnmyjkSJfyBGHH25zIiqjX2H0aLtCOPXU3duuvBJatIBhw2xkVCaK/Nt3T5NpuNFmNP8lymsKnFDBsTjnCsnNhQkTrNlm9Gg48sji26a7d4eZM/fsf6hoH30E06fDyJF79l1Urw533WUVYJ98cs+mrUwxa5Y1ox11VKIjqRglXimoau8oN08IzsVZbq5NTBs3zr70r77a+hSKysmBL76wkt3x8uijlgAGDdr7tXPOga5drQP8p5/iF0Oymj3b/hukypDTWMKsvHaNiNQt9PxAEfElxp2Ls0jJi6FDYZ99rMOyOIX7FeLhp5/sKuCcc+Dgg/d+XQTuu8/KbzzwQHxiSFZffmkjjFKh0F1YYeYpDFbVjZEnqroBGBy/kJxzAAccYL/Af/oJzjvPFugpTps2Vh4jXklh4kTYuHHv/ozCcnKsE/zuu6E062CppvbIpdmz7T5d+hMgXFKoGqyHAICIVAX2iV9IzrmISBG1IUNK3qdqVZvdHK+kMHq0jYA6/vjo+919tzV33Xtv+HP//e/QrFnqdlLPmgU1a0KHDomOpOKESQqvAs+KyIkiciK27sGrMY5xzlWA666DF16A44quWVhETg58+CFs2FCx779ggdVWysuL3WaelWVNTI8/Dj//HPvc27fDqFGwejW8/36FhFvpZs+2q7nI+hnpIExS+D/gTWzFtSHAG8DQeAblnDMHHABnnRX7Czknx5ph3n23Yt9/zBj7JXzZZeH2HzLExuw/91zsfSN1nCA1S4D/8IOVsUin/gQIt/LaTlUdo6rnBrdHVXVHZQTnnAunSxcrfVGRX67ffw///CcMGGB1lsLo3dvmLYwZE3vf0aPh0ENtnkWkbT6VzJljs5nTqT8Bwl0pOOeSXK1a0KlTxSaFiROtk/vqq8MfI2JNTe++G71JKFLHafBg6NnT4k61DudZs2yIcLduiY6kYnlScC5NdO9u7f+x1ncOa9IkmzDXuXPpjrvsMlt1LlJiuziROk6DB1vcX30Fq1aVL97KNmsWtG1rI7/SSeikICK14hmIc658cnIsISxYUP5zffedrf987rmln5R10EFw4YXW9FTc6myROk5nnGF1nOI9zyIetm+H995Lv6YjCDd57TgRWQp8FDxvJyKPxD0y51ypRL6gKuLL9aWX7Ivv3HPLdnxeHvz4o5X1LipSxykyzLZ1a+tQT6V+hffft+G36dbJDOGuFO4HTgHWA6jq+0DPeAblnCu9hg2tk7ciksKkSVb8rlOnsh3fpYuN3R89eu++gqJ1nKpUsYSWSlcKkVgzNSmgqmuKbPLRR84loe7d7Rd3eTptv//eqrP271/2ej6RDufFi/ccJrtkSfF1nLp3h6VLbZWyVDBrliXNxo0THUnFC5MU1ojIcYCKSHURuRFYFue4nHNlkJNjX6yFF+cpralTYdu2sjcdRVx0ka25UHh46qOPFl/HKfKL+513yveelUHVkkI69idAuKSQB1wDNAK+ANoHz51zSSby5TpzZtnPMWmSzR8o7xKb++0Hl15qQ1vXr7c+hqeeKr6OU+fONis4FfoVVq2Cr79Oz6YjiL6eAgCq+i1wcSXE4pwrp+bNrdO2rGUjfvwRXn3VhooWV6a7tPLy4OGHbVWyunVtecri6jjVrAnZ2anRr5DO/QkQIimIyN+L2fw9UKCqUyo+JOdcWYlYZ/Py5WU7/uWXYcsW60+oCG3a2JfnmDGWrNq0KbmOU04O/O1v9v41alTM+8fDrFn2WVq3TnQk8RHmt0ANrMloRXBrCzQGBolIhlVPdy75lScpTJpko5gq8lfwkCGwcqXNnxgypOTO6+7dbZW5goKKe+94mDXLEltFXEklozAfqy3QW1UfVNUHgZOAo4Gzgdx4BuecK70WLWwVtjCVSgv76Sf473+t0mnhJTfLq39/60OoXRsGDix5v8gVRDL3K6xfD8uWpW/TEYRLCgcC+xV6Xhs4KCiKF3VCvYjcICIfisgSEXlGRGoUef1+EVkU3JaLyMaSzuWcC6dFCxshs3Jl6Y577TVLDBXVdBSx77426mjMmOglIRo0sHUbkrlfITI6Kp2TQsw+BeBeYJGIzAAEm7h2l4jUBl4v6SARaQRcC7RS1Z9FZCJwIfBkZB9VvaHQ/r8D0mipCucSo0ULu1++3Nrww5o0CerVi72YTlmcc064/XJy4PnnrfpoMjbPzJplo6RKWw8qlYQpnT0eOA54EXgByFHVx1R1s6reFOPwakBNEakG1AK+jLLvAGwBH+dcOTRvbvel6Vf45Rf4z39s7YZELhiTk2MLBS1L0plQs2bZLO+aNRMdSfyEzcVbgK+ADcBRIhKzzIWqfgGMAj4Pjv1eVfOL21dEjgCaYov5OOfKoU4d+NWvSpcUpk2z4nUV3XRUWpEJYcnYr7B5s3WCp+uktYgwBfGuBN4GXgP+ENyPCHHcgUA/7Mv+UKC2iJTUzXQhMKmkxXtE5CoRKRCRgnWlWRXcuQzVsmXpksKkSTbM8sQT4xdTGEceCQcfnJz9Cq+8YqOj+vRJdCTxFeZK4TqgM/CZqvbG2v3DdAifBKxS1XWqug2YjDVDFedCojQdqepYVc1W1ewGDRqEeGvnMltphqVu3QpTpsCZZ1oJikQSsSakZEwKkyZZZ3iPHomOJL7CJIUtqroFQET2VdWPgJYhjvsc6CoitUREgBMppmaSiByNjXCq4NVlnctcLVrAunXWPh/L9OmwcWP5ax1VlJwcKyXxxReJjmS3n3+2mlBnnWXLnqazMElhrYjUxTqap4nIFOCzWAep6hxgErAAWBy811gRuVNEziy064XAv1VTbTE+55JXZATSihWx95061ZbzzE2SWUeR4Z7J1K+Qn299CsmSOOMpTO2js4OHI0RkOnAA8GqYk6vqHcAdRTbfXmSfEWHO5ZwLr/Cw1C5dou87d64NsUyW0hLt2lmSmj0bzj8/0dGYSZPgwAOhd+9ERxJ/Ua8URKSqiHwUea6qb6nqS6q6Nf6hOefKqmlTm5Ucq19h61YrnpdM4+6rV4euXZOnXyEyXLdfv8QO160sUZNCMBroYxE5vJLicc5VgH32scQQKyl8+KF96WVnV05cYeXkwKJFxa/xXNneeMMWHsqEpiMIX+biQxF5Q0ReitziHZhzrnzCjECaN8/uky0p5ObarOZhwxIdiTUd7b//7uVD012YfvThcY/COVfhWrSAt96yOkglVSYtKLC28mbNKje2WLp3hxtvhFGjrI/hqqsSE8e2bTZc94wzrIZTJghT5uItYDVQPXg8DxtR5JxLYi1a2IiZr74qeZ+CArtKKOtazPF0991w6qlwzTXlW0muPGbMgO++y5ymIwg3o3kwNrT00WBTI2x4qnMuiRUegVScLVtg8eLkazqKqFoVnnnGZjn37w+fxRwIX/EmTbKS36ecUvnvnShh+hSuAboDmwBUdQXQMJ5BOefKL1ZSeP992L49eZMC2BKeU6bYKKmzzrIrn8qyYwe88IKVtUjnAnhFhUkKvxQeghpUPPWJZs4luUaN7MuspKQQWeEsmYajFqdlS7tieP99uPxy6yOpDDNn2qzwTGo6gnBJ4S0RuQUrgX0y8Bzwn/iG5ZwrrypVrIx2tKTQsCE0bly5cZXFaafBPffAc8/ByJGV856TJllSPe20ynm/ZBEmKQwD1mGlKq4GXgZui2dQzrmKEW1Y6rx5ydvJXJwbb7TlPIcPt7kDYT34IPz2t6V7r507YfJk6+jeb7/Y+6eTMEnhLOApVT1PVc9V1XFep8i51NCihS3LuX37ntt//NEWsknm/oSiRGDsWGsWGzUq3DE//wy33w4PPwyvl7hO5N7efddGbWVa0xGESwpnAMtF5B8i0jfoU3DOpYAWLSwhrFq15/ZFi+zXcLL3JxRVsyYMHmzrSX/6aez9n33WKsDWqQNDh9pnDuP5521WeN++5Ys3FYWZp3A5cBTWlzAAWCkij8U7MOdc+ZU0Aikyk7lTp8qNpyJceaX1lzz6aOx9x4yBo4+GRx6BhQutwzoWVetPyM21mcyZJtRynMEiOa8A/wbmY01KzrkkV1JSKCiwZphf/aryYyqvRo1sQaDHH7e6TSVZuBDmzIG8PLjoIujQAW69NfoxAC++CGvWJH5p0kQJM3ntNBF5ElgB9AceAw6Jc1zOuQpQrx4cdFDxSSHVmo4KGzIEvv3WmnlKMmaMNTddeqldWdx7r02Ae/jhko9ZuhQuuww6doQLLqj4uFNBmCuFS7EZzC1V9deq+rKqbo91kHMuORQdgbRxoz1PpU7mok480WY6jxlT/OubNsE//wkXXmi1ncAK2uXm2pDWjcUsKLxhg5XHrlXLrhYyacJaYWH6FAao6ouq+guAiOSISJRc65xLJkWTwoKgclkqJ4UqVaxZaOZMWLJk79efftpmPw8Zsuf2e+6xL/+7795z+/btdmXw2Wd29XHYYfGLPdmF6lMQkQ4icp+IrAb+CHwU4xDnXJJo0QLWrt1dIiIykzmVkwLAr39tlUuLXi2owujR1gRU9DO2b29zHR54wPoNIoYOhWnT7Lju3eMeelIrMSmISAsRuSNYee1B4HNAVLW3qj5YaRE658ol0tn8ySd2P2+eLcBTr17iYqoI9evDeefBU0/ZvIuId96xq4chQ4qfmPfHP9r97cHCwBMmwP33w+9+B4MGxT/uZBftSuEj4ASgr6rmBIlgR+WE5ZyrKEVHIEXKZaeDvDxbna3wUNPRo20o6YABxR9zxBGWACZMgHHjbK2GE06Av/ylcmJOdtGSwjnAV8B0ERknIicCKTIh3jkXcdRRdr98uY3YWb06fZLCccfBMcdYIlC1AnbPPWcjjmrXLvm4W26xCqxXXWW1nyZOzIz1l8MoMSkEncsXAkcD04HrgYYiMlpEcisrQOdc+dSubV98y5enTmXUsESsmWjhQmsWe/JJK7Odlxf9uAMPhLvugoMPttLcqd6UVpGkNGWMRORA4DzgAlU9MW5RRZGdna0Fkb9s51woJ54IP/1kawMMH25DMg84INFRVYxNm+DQQ22y2ezZ9vjtt8Mdu3OnjWTKBCIyX1VjXiOW6p9DVTeo6thEJQTnXNlEhqUWFNjjdEkIYP0HAwdah/PKlXsPQ40mUxJCafg/iXMZoEULW2t4xoz0aToqLNJc1KABnHNOYmNJdV7x1LkMEBmB9P336dPJXFj79nDFFVbgb999Ex1NavOk4FwGiCQFSM+kADB+fKIjSA/efORcBmjSBKpVszb0Dh0SHY1LZn6l4FwGqF4dmjWzhWOijd93Lq5JQURuAK4EFFvj+XJV3VJkn/OBEcE+76vqRfGMyblMdeed3t7uYotbUhCRRsC1QCtV/VlEJgIXAk8W2qc5cDPQXVU3iEjDeMXjXKbL1PUBXOnEu0+hGlAzWNe5FvBlkdcHAw+r6gYAVf0mzvE455yLIm5JQVW/AEZh1VW/Ar5X1fwiu7UAWojIbBF5T0ROjVc8zjnnYotbUghKYvQDmgKHArVFZGCR3aoBzYFewABgnIjULeZcV4lIgYgUrFu3Ll4hO+dcxotn89FJwCpVXaeq24DJwHFF9lkLvKSq21R1FbAcSxJ7CEprZKtqdoMGDeIYsnPOZbZ4JoXPga4iUktEBDgRWFZknxexqwREpD7WnPRpHGNyzjkXRTz7FOYAk4AF2HDUKsBYEblTRM4MdnsNWC8iS7Hy3Dep6vp4xeSccy66UpXOTgZeOts550ovLqWznXPOpbeUu1IQkXXAZ2U8vD7wbQWGk0oy9bP7584s/rlLdoSqxhypk3JJoTxEpCDM5VM6ytTP7p87s/jnLj9vPnLOObeLJwXnnHO7ZFpSGJvoABIoUz+7f+7M4p+7nDKqT8E551x0mXal4JxzLoqMSQoicqqIfCwin4jIsETHEy8i8riIfCMiSwptO0hEponIiuD+wETGGA8icpiITBeRpSLyoYhcF2xP688uIjVEZK6IvB987j8E25uKyJzg7/1ZEdkn0bHGg4hUFZGFIjI1eJ72n1tEVovIYhFZJCIFwbYK+zvPiKQgIlWBh4HTgFbAABFpldio4uZJoGgJ8mHAG6raHHgjeJ5utgO/V9VWQFfgmuC/cbp/9l+AE1S1HdAeOFVEugL3APer6lHABmBQAmOMp+vYs6Zapnzu3qravtAw1Ar7O8+IpAB0AT5R1U9VdSvwb6ysd9pR1beB74ps7gdMCB5PAM6q1KAqgap+paoLgsc/YF8UjUjzz67mx+Bp9eCmwAlY7TFIw88NICKNgT7AY8FzIQM+dwkq7O88U5JCI2BNoedrg22Z4mBV/Sp4/DVwcCKDiTcRaQJ0AOaQAZ89aEJZBHwDTANWAhtVdXuwS7r+vT8ADAV2Bs/rkRmfW4F8EZkvIlcF2yrs7zxuazS75KSqKiJpO+RMRPYDngeuV9VN9uPRpOtnV9UdQPtggaoXgKMTHFLciUhf4BtVnS8ivRIdTyXLUdUvgjXtp4nIR4VfLO/feaZcKXwBHFboeeNgW6b4n4j8CiC4T8u1sEWkOpYQ/qmqk4PNGfHZAVR1I1aCvhtQN1gbHdLz7707cKaIrMaag08A/kb6f+7IUseRNe1fwJrHK+zvPFOSwjygeTAyYR/gQuClBMdUmV4CLgseXwZMSWAscRG0J48HlqnqXwu9lNafXUQaRJawFZGawMlYf8p04Nxgt7T73Kp6s6o2VtUm2P/Pb6rqxaT55xaR2iJSJ/IYyAWWUIF/5xkzeU1ETsfaIKsCj6vqyASHFBci8gy2ml194H/AHdgKdxOBw7EKs+eratHO6JQmIjnATGxBp0gb8y1Yv0LafnYRaYt1LFbFfuRNVNU7RaQZ9gv6IGAhMFBVf0lcpPETNB/dqKp90/1zB5/vheBpNeBfqjpSROpRQX/nGZMUnHPOxZYpzUfOOedC8KTgnHNuF08KzjnndvGk4JxzbhdPCs4553bxpOCSjoioiPyl0PMbRWREBZ37SRE5N/ae5X6f80RkmYhMr4BzPVbeAo4i0qRw5VznSuJJwSWjX4BzRKR+ogMprNBM2TAGAYNVtXd531dVr1TVpeU9j3NheFJwyWg7trzgDUVfKPpLX0R+DO57ichbIjJFRD4VkbtF5OJgrYHFInJkodOcJCIFIrI8qKETKSp3n4jME5EPROTqQuedKSIvAXt9MYvIgOD8S0TknmDb7UAOMF5E7ivmmJsKvU9k/YMmIvKRiPwzuMKYJCK1gtdmiEh2EOOTwXstFpEbgtfbi8h7wfleiNTSF5FOYussvA9cU+j9i/2szoEnBZe8HgYuFpEDSnFMOyAPyAIuAVqoahestPLvCu3XBKsX0wcYIyI1sF/236tqZ6AzMFhEmgb7dwSuU9UWhd9MRA7F6vefgK1l0FlEzlLVO4EC4GJVvanIMblA8+D92wOdRKRn8HJL4BFVzQI2Ab8p8vnaA41UtY2qHgM8EWx/Cvg/VW2Lzei+I9j+BPC7YK2FwqJ9VpfhPCm4pKSqm7Avu2tLcdi8YF2FX7Dy0fnB9sVYIoiYqKo7VXUF8ClWVTQXuDQoQT0HK8PcPNh/rqquKub9OgMzVHVdUK75n0DPYvYrLDe4LQQWBO8deZ81qjo7ePw0drVR2KdAMxF5UEROBTYFSbOuqr4V7DMB6BnUQ6obrK8B8I8iMZT0WV2G89LZLpk9gH1xPlFo23aCHzMiUgUovNxi4Ro3Ows938mef+tFa7soINiv6tcKvxDU1dlctvCLJcCfVfXRIu/TpIS4dj9R3SAi7YBTsCui8ymmiS1kDHt9VufArxRcEgsKek1kzyUVVwOdgsdnYiuNldZ5IlIl6GdoBnwMvAYMCcpvIyItgiqU0cwFjheR+mJLvg4A3opxzGvAFWLrPiAijcTq4gMcLiLdgscXAbMKHxh0vFdR1eeB24COqvo9sEFEegS7XQK8FZTR3hgUCgS4uEgMpf2sLkP4lYJLdn8Bflvo+ThgStB5+ipl+xX/OfaFvj+Qp6pbROQxrIlpgYgIsI4YSxqq6lciMgwr1yzAf1U1asliVc0XkSzgXXsbfgQGAjuw5HSNiDyOdWqPLnJ4I+CJ4AoJ4Obg/jKsb6QW1sR0ebD9cuBxsQVX8gudp9Sf1WUOr5LqXBIImo+mqmqbBIfiMpw3HznnnNvFrxScc87t4lcKzjnndvGk4JxzbhdPCs4553bxpOCcc24XTwrOOed28aTgnHNul/8PImyyaNBZD6kAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "bandwidth = 100 # Moving average bandwidth\n",
    "total_clipped = np.zeros(int(epis_count)-bandwidth)\n",
    "total_rew = np.zeros(int(epis_count)-bandwidth)\n",
    "for i in range(int(epis_count)-bandwidth):\n",
    "    total_clipped[i] = np.sum(tot_clipped_reward[i:i+bandwidth])/bandwidth\n",
    "    total_rew[i] = np.sum(tot_reward[i:i+bandwidth])/bandwidth\n",
    "t = np.arange(int(epis_count)-bandwidth)\n",
    "belplt = plt.plot(t,total_rew[0:int(epis_count)-bandwidth],\"r\", label = \"Return\")\n",
    "plt.legend()#handles[likplt,belplt])\n",
    "print('Running after %d number of episodes' %epis_count)\n",
    "plt.xlabel(\"Number of episode\")\n",
    "plt.ylabel(\"Average Reward per episode\")\n",
    "plt.show()\n",
    "likplt = plt.plot(t,total_clipped[0:opt.num_episode-bandwidth],\"b\", label = \"Clipped Return\")\n",
    "plt.legend()#handles[likplt,belplt])\n",
    "plt.xlabel(\"Number of episode\")\n",
    "plt.ylabel(\"Average clipped Reward per episode\")\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Accumulated average reward after 1000 episodes of game Assault\n",
    "|![](../img/Assault.png)|![](../img/Assault-clipped.png)|\n",
    "|:---------------:|:---------------:|\n",
    "|Average reward|Average clipped reward|"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
